Competitive intelligence · March–September 2026
CALI
Competitive Agent Longitudinal Intelligence
- participants across seven AI agents in the September 2026 readout
- 264
- recruitable pool for fresh or longitudinal panels
- 2.25M+
- months tracked month over month since March 2026
- 7
- of monthly CX allocation for the 180-participant/year baseline
- ~3%
Case Study: CALI — Competitive Agent Longitudinal Intelligence
Designed and led CALI, a longitudinal competitive intelligence system that tracks how customers actually experience, trust, and adopt AI shopping agents enabling faster, lower‑risk product decisions grounded in live market behavior.
Context & Problem
The problem
- AI shopping agents are evolving rapidly across competitors.
- Traditional research (one‑off usability tests, post‑launch surveys) is too slow and too fragmented to inform strategy.
- Leadership lacked a continuous, customer‑grounded benchmark for value, trust, and adoption across AI agents.
Why this mattered
- High uncertainty, high visibility product space
- Late‑stage validation increased risk and rework
- Decisions were often based on assumptions, anecdotes, or lagging indicators
My role
- Principal researcher and architect of the CALI methodology
- Defined the measurement framework, sampling strategy, and reporting cadence
- Partnered with product, design, and leadership to align outputs to decision‑making needs
Scope
- Competitive intelligence
- Longitudinal behavioral research
- Trust, adoption, and AI‑human interaction measurement
The Solution: What CALI Is
CALI is a flexible, longitudinal research system that:
- Collects structured qualitative signals weekly
- Applies a stable coding and measurement framework
- Produces monthly and quarterly competitive intelligence on AI agents
What it delivers
- A monthly pulse (current behavior + composite scores)
- A quarterly synthesis (trends, narratives, implications)
- A forecast of likely adoption and trust trajectories
How It Works (Method Overview)
Sampling
- Flexible design:
- Fresh participants each wave or
- Longitudinal panel over time
- Large recruitable pool (2.25M+)
Data collection
- Participants walk through their last three real interactions with AI agents
- Agents include competitive systems (e.g., Rufus, Sparky, Magic Apron)
- Participants reference actual chat history (grounded recall)
Measurement
- Structured, repeatable battery assessing:
- Behavior
- Experience quality
- Trust and adoption outcomes
Synthesis
- Monthly research memo on a predictable cadence
- Centralized archive for continuity and learning
What CALI Measures (Core Framework)
Core dimensions
- App value (Was it worth using?)
- Acceptance (Will I use it again?)
- Trusting stance toward AI
- Information credibility
- System quality (reliability, flexibility)
- Social influence
- Organizational assurance
Optional high‑value dimensions
- Privacy comfort
- Control and editability
Composite score
- All dimensions compiled into a service composite score
- Calculated as an unweighted mean
- Enables clear, high‑level comparison across agents
Output Cadence & Cost Awareness
Reporting cadence
- Monthly pulse: lightweight signals + composite scores
- Quarterly synthesis: trends, patterns, implications
- Recorded readouts + centralized archive
Resourcing
- Baseline model:
- 15 participants/month
- 3 services
- 180 total participants/year 20k
- ~3% of monthly CX allocation
- Easily scalable to biweekly or weekly
This approach allows us to see how customers actually use these services, where they succeed, and where friction occurs reducing reliance on assumptions and late‑stage validation. Over time, the same longitudinal signals create a clear benchmark, enabling us to directly plot our performance against the competition and identify areas of differentiation, parity, and risk. The result is faster, more confident product decisions grounded in live competitive reality.
Impact & Value
This is the most important section.
What CALI enables
- Immediate diagnosis of why peaks and valleys occur
- Early detection of trust, adoption, or quality risks
- Reduced reliance on assumptions and late‑stage validation
Over time
- Creates a living benchmark against competitors
- Identifies differentiation, parity, and risk areas
- Drives faster, more confident product decisions
Bottom‑line impact
Replaced episodic research with continuous competitive intelligence—grounded in real customer behavior.
CALI in practice: the September 2026 report
Seven months in, CALI's September readout covered 264 participants across seven AI agents: retail assistants (Walmart Sparky, Amazon Rufus, Amazon Alexa, and KSA) and general-purpose models (ChatGPT, Claude, and Gemini), tracked month over month since March.
How each wave runs
- Last 3 interactions: participants walk through their three most recent real conversations with an agent.
- TrAAIT: each interaction is scored with the Theory of Trust and Acceptance of Artificial Intelligence Technology instrument.
- Customer segment battery: participants are placed into shopper segments, so results can be read by who is shopping.
- Report: a monthly readout of scores, segments, and what changed.
Who we heard from
Amazon & Walmart shoppers: 163 participants, 467 interactions
- Average age 37; about 61% female; the most common income bracket for both women and men is $100K–$124,999
- Quality Seeking Savers are the largest segment (about 45%), and Walmart skews further toward them (52% vs Amazon's 37%)
- Reward Focused Cooks (36% vs 24%) and Wellness Obsessed Shoppers (23% vs 15%) lean Amazon; Budget Strapped Shoppers lean Walmart (7% vs 3%)
General-purpose models (ChatGPT, Claude, Gemini): 92 participants, 251 interactions
- Average age 37; nearly two-thirds female; $40K–$79,999 is the most common income band (44%)
- Quality Seeking Savers lead on every platform (about 46% overall, 50% on Gemini)
- Claude users skew Wellness Obsessed (35% vs ChatGPT's 20% and Gemini's 17%); ChatGPT leads in Reward Focused Cooks (27%)
KSA shoppers: 9 participants, 27 interactions
- Reward Focused Cooks are the largest segment (50%), with Quality Seeking Savers close behind (40%)
- A small debut panel, so these findings are directional only
Service Directional Index
What's working
- Future-use intent is universally strong: ChatGPT and Gemini scored a perfect 5.0 on likelihood to use in every month, with Amazon Alexa close behind at 4.8–4.9
- AI-native platforms lead on satisfaction and ease of use (4.6–5.0), while retail assistants lead on trust and institutional confidence
What needs attention
- Trust and confidence are the lowest-scoring metrics on every platform, every month. No platform has solved this yet.
- Vendor support is declining industry-wide. AI-native platforms score as low as 2.7, a signal that users want greater organizational accountability.
- Outlier participants (scoring 1–2) skewed means by up to 0.7 points in nearly every study.
KSA's debut
- Highest confidence score (4.5) of any retail AI assistant across all platforms and months, ahead of Amazon Alexa and Walmart Sparky
- Ease of use and reliability both scored 4.5; core users rated satisfaction, accuracy, and reliability near a perfect 5.0
- Vendor support (3.6) and information currency (4.1) are the lowest scores and the priorities for improvement
Reading the outliers
Two participants scored KSA between 1 and 2, pulling averages down by up to 0.7 points. Without them, most scores reach 4.8–5.0. CALI's interviews showed the two low scores had different causes.
"I'm not very satisfied because it just needs a little bit more training… the discounted items were discounted a month ago — there's no discount when you go in the store. It just cannot switch between prompts."
Legitimate product gaps: outdated discount data, poor multi-prompt handling, and no in-store navigation. This participant still expressed long-term confidence in KSA.
"I tried two apps, the Kroger app and the Smith's app, and tried to do their AI for shopping… I couldn't get it to work either time, so it left me feeling frustrated."
A technical failure, not a perception problem: the app didn't work during the session. This participant showed genuine interest in AI shopping features.
The same low scores pointed to two different fixes: better data and conversation handling for one, app reliability for the other. That is the diagnosis a one-off survey score can't give.
Source: Stevens AF, Stetson P. Theory of trust and acceptance of artificial intelligence technology (TrAAIT): An instrument to assess clinician trust and acceptance of artificial intelligence. J Biomed Inform. 2023;148:104550. doi:10.1016/j.jbi.2023.104550