Wizard-of-Oz research memo · June 2026
Meal Planning Agent Study
A Wizard-of-Oz blind preference test of three AI meal-planning agents, with real grocery carts
- participants planning tonight's dinner
- 12
- AI agents, blind and rotated
- 3
- in true preference: Borda count 22, 22, 19
- Tie
- acceptance: no agent crossed the line
- 3.52
The opportunity: help people get from meal inspiration to a relevant, actionable grocery shop.
Three AI meal-planning agents, Cooklist, KSA, and Alpha, were tested blind with twelve shoppers who planned a real dinner and built a real grocery cart. The survey crowned one winner. Participants' own rankings told a different story, and the gap between the two is where the most useful findings were.
The study
- 12 participants planned tonight's dinner using 3 AI agents
- Agents were blind, with order rotated across positions
- Participants thought aloud while researchers followed their lead
- Participants added real ingredients to a cart
- Each agent was rated on a structured battery, then participants picked an overall #1, #2, and #3
"Think about tonight — who you're feeding, schedules, what's in the fridge. Plan a meal and add ingredients to your cart."
That was the only prompt participants were given.
Tool B (Alpha) opened as an assistant over a grocery website; Tool C (Cooklist) was a standalone assistant page with suggested prompts.
Alpha mid-session: a recipe, quick steps, and its ingredients as products ready to add to the cart.
Tool A (KSA) ran inside a mobile app.
A caveat on counterbalancing
Rotation was close to balanced, but not perfect:
| Agent | First | Middle | Last |
|---|---|---|---|
| Cooklist | 4 | 3 | 4 |
| KSA | 5 | 3 | 3 |
| Alpha | 3 | 5 | 3 |
- KSA may have a slight primacy boost
- Alpha may carry a middle-position penalty
- Cooklist is the cleanest read
- 2 sessions were incomplete because of technical issues
Survey results
| Rank | Agent | Composite mean |
|---|---|---|
| 1 | Cooklist | 3.97 |
| 2 | Alpha | 3.79 |
| 3 | KSA | 3.63 |
Cooklist led on 10 of 12 survey metrics. The battery used a 5-point scale (Not at all → Extremely) across five TrAAIT constructs:
- Trusting Stance: baseline attitude toward AI before use
- Tool Value: ease of use and task satisfaction
- Information Confidence: accuracy, quality, and satisfaction with the information returned
- System Quality: reliability and adaptability of the tool
- Acceptance: intent to actually use it in the next 3 months, for household shopping
Together they track the funnel from "do I trust AI?" to "did this tool deliver?" to "will I adopt it?"
Where Cooklist wins
Information Confidence is Cooklist's moat:
| Metric | Cooklist | KSA | Alpha |
|---|---|---|---|
| INFC2 | 4.82 | 3.73 | 4.45 |
| INFC3 | 4.55 | 3.73 | 4.27 |
| INFC4 | 4.18 | 3.82 | 3.91 |
| Combined | 4.52 | 3.76 | 4.21 |
Users trust Cooklist's information, even users who don't love the tool overall.
Where KSA loses
KSA finished last on 5 of 5 content and quality metrics:
- Lowest Information Confidence
- Lowest System Quality
- Trust and polish are the gaps
But hold that thought.
The plot twist: survey ≠ preference
| Agent | Survey rank | #1 picks | #3 picks |
|---|---|---|---|
| Cooklist | 1 | 4 | 2 |
| Alpha | 2 | 3 | 3 |
| KSA | 3 | 4 | 5 |
KSA tied Cooklist for the most #1 picks, and got the most last-place finishes.
There's an asterisk on Cooklist's win count. One of its #1 picks came from a session where KSA was the only agent the participant didn't experience, because technical issues kept it from loading. With that session removed from the tally, KSA has the most #1 picks.
A Borda count reveals a tie
Scoring 3 points for a #1 pick, 2 for #2, and 1 for #3:
| Agent | Survey rank | Borda count | Personality |
|---|---|---|---|
| Cooklist | 1 | 22 | The consensus choice |
| Alpha | 2 | 22 | The strong runner-up |
| KSA | 3 | 19 | The polarizing agent |
Cooklist and Alpha are effectively tied in true preference. The three agents have distinct personalities:
- Cooklist, the consensus choice: broadest appeal, rarely rejected. The "safe favorite."
- Alpha, the strong runner-up: the most #2 picks. "Good enough" for everyone.
- KSA, the polarizing agent: loved or hated, almost never #2. A niche it owns.
The universal weak spot
No agent crossed the line on acceptance:
- Tool Value: about 4.0
- System Quality: about 4.0
- Acceptance: 3.52
Users say these tools work, but won't commit to using them weekly. Whoever solves acceptance wins long-term retention.
When the top pick isn't the top score
Two participants picked a #1 that wasn't their highest-rated agent.
One participant evaluated the tools through two distinct lenses. As a cook who knows what she wants to make, she picked KSA, even though it didn't score highest on her battery: its strength was getting out of the way and executing on a clear intent. As a cook who doesn't know what to make, she picked Cooklist, for its ability to inspire and guide a decision from scratch.
The other was a pragmatic shopper. She saw KSA as an improvement on how she shops today, which drove her #1 pick. She felt there was a learning curve with agents overall, and that Alpha didn't offer enough detail to be useful.
What worked well in KSA
- Product variety per ingredient. "I really like being able to scroll through and see, compare different ingredient costs before selecting one to put in my shopping cart. That's what I do at the grocery store."
- Inline cart-building, with no tab-switching. "It's clearly just in the app, everything can be done there. I'm not opening new windows or new tabs or anything like that."
- Stock visibility. Seeing in-stock vs. out-of-stock meant not being left "wondering why can't I see my brand."
- Efficient for known-recipe shopping. "It would be more quicker for me to just say, hey, let's do a 12-inch pizza."
- Quantity guidance per ingredient. The products view showed "how much I need for this recipe of each of these items… okay, I need 2 tablespoons of olive oil."
- Conversational recovery works. Several participants refined results by typing follow-ups in "Ask Me Anything," and once they learned the pattern, satisfaction jumped: "Now that I know how to do it, I think me using it for the second time… would be much easier."
What didn't work well
- Mode ambiguity: recipe tool or grocery tool? "At first I thought I was just looking for recipes, kind of like I would in ChatGPT, but by the time I was done, it made me feel like it was kind of like a grocery store app."
- A broken cart mental model. There was no persistent, visible cart indicator, so users lost trust that their actions registered. Even a participant who loved KSA asked for "a link or something to look at the cart."
- Irrelevant results to clear prompts. A request for a "family-friendly chicken meal" returned chicken broth ("That's not a meal") and chicken tenders. Fruit for tacos returned watermelon, grapes, and blackberries, but no pineapple or mango. Another suggestion missed the meal occasion entirely: "not really for lunch."
- Stock-outs dominated the experience. One participant had to switch her entire dinner plan twice: "So many items not available… at this point, I would most likely create a dinner versus using the AI option."
- Limited brand and option surfacing. "There are way more options than that, 'cause I've bought other brands before… you're kind of having to spend more time explaining." Ironic, given product variety is also a KSA strength: depth is inconsistent across categories.
- No way to save recipes. "I would like to save that… I don't see a way to save that… Maybe I have to take a screenshot."
- No memory, profile, or history. "I would like a history so they can know what recipes I usually shop ingredients for."
- No recipe images. A participant asked for images, especially to tell baked from fried empanadas.
- No reviews, ratings, or social proof. "Being able to click on a recipe and see reviews or some kind of feedback on the quality would be a nice feature."
- No ingredient or nutrition detail. A participant wanted to click a sauce to check its ingredients for a picky eater, and couldn't.
- No dietary or preference profile. One participant wanted to permanently store "my family hates red and green bell peppers" and dairy sensitivities. There was no way to set persistent constraints.
Product implications
- Balance choice with a clear route to action. Offer relevant recipes and product alternatives, but keep selection, comparison, and cart status simple.
- Make the cart visible and continuous. A persistent cart is how users know their actions registered.
- Don't repeat what users already rejected. Every repeated suggestion costs trust.
- Tailor to the occasion and the household. Meal occasion, dietary needs, and dislikes should be remembered, not re-explained.
- Let users steer the level of help. Some shoppers want inspiration and options; others want one fast recommendation. The experience should let them choose, rather than assume one level of detail works for everyone.
- Solve acceptance. Tools that "work" still aren't used weekly. That gap is the long-term retention lever.
Cooklist set the baseline for trusted information, KSA showed the value of fast execution when intent is clear, and Alpha was the broadly acceptable runner-up. The opportunity is to unite those strengths in an assistant that makes meal planning relevant, choices manageable, and the next step unmistakable.
Scope
These results are formative evidence from 12 sessions, not a population-wide estimate. Two sessions were incomplete because of technical issues, including one where KSA wouldn't load, and counterbalancing was close but not perfect. Those constraints shape what to validate next.
Sources
Stevens AF, Stetson P. Theory of trust and acceptance of artificial intelligence technology (TrAAIT): An instrument to assess clinician trust and acceptance of artificial intelligence. J Biomed Inform. 2023;148:104550. doi:10.1016/j.jbi.2023.104550