Updated November 2026
With all the noise around AI, it's easy for a shiny idea to find its way onto the roadmap. Before I ask engineering to build it, I want to see what it would improve for the person doing the work.
I think every product manager working on AI features should be able to put a small, testable version in front of users before handing it to engineering. AI features come with a cost. I'd start by watching users do the task today, then build the smallest version of the idea myself and put it in front of them.
In short, I'd build an AI feature when a small prototype shows three things on the real task. It saves time once review and corrections are counted, its output meets a quality bar users would accept, and people can still finish when it has no good answer. The running cost per task also has to make sense against a simpler alternative.
Below are the five decisions I'd work through, using an ecommerce quiz builder where the idea on the table is AI-generated product recommendations.
The example: a merchant setting up a product quiz
A merchant setting up a product quiz writes the questions, defines the results, links products to each result, previews the quiz and publishes it.
The AI idea sits in the product-linking step. For each result, such as "dry skin, budget under £30", the merchant decides which products to show and attaches them from the catalogue. I've written separately about why a quiz is a website asset for your product offer.
1. Find the friction in the current workflow
I'd watch a merchant set up a real quiz. Are they struggling to decide which products to recommend? Or do they already know and just need a quicker way to link products to results?
That distinction tells me whether to prototype AI suggestions or a simpler setup flow. If a merchant spends ages deciding which products to recommend, AI suggestions might help. If they already know which products they want and get stuck linking them to quiz results, I'd try a better product picker first.
While watching, I'd note:
How long linking products takes for each result, compared with the rest of setup.
Where the merchant goes to decide, such as their store in another tab, a bestseller report or a spreadsheet.
Whether they hesitate over which product to choose or over finding a product they have already chosen.
What they do when no product fits a result.
Observation is one part of the wider customer evidence I look at before a product decision.
Watching first matters because AI performance depends on the task. A field experiment published in Organization Science gave 758 Boston Consulting Group consultants access to GPT-4. On 18 tasks within the AI's capabilities, they completed 12.2% more tasks and worked 25.1% faster. On one task chosen to sit outside those capabilities, they were 19 percentage points less likely to reach a correct answer. Only the real task with a real catalogue will show which side product selection falls on.
2. Prototype one step and compare it with a non-AI option
I'd use Claude Code to prototype one step. Suggest products for a quiz result and let the merchant approve, edit or reject them. Show it first and get feedback early on.
The smallest clickable version has three screens:
Suggestions for one result
The merchant sees three suggested products, each with a short note on which quiz answers it matches, and buttons to approve, swap or remove it.
Swap
Opens a catalogue search to replace a suggestion.
No good match. When nothing fits, the screen says so and opens the search straight away.
For the comparison, I'd build a searchable product picker. It has one search box, filters by collection and tag, and multi-select to attach several products at once. It is a simpler feature, but can actually solve the problem in the exact same way and this is how we will test it.
Both versions use the same quiz and the same catalogue, ideally a copy of the merchant's own. Each merchant links products for the same results in both versions. I'd switch the order between merchants, so neither version always benefits from practice, and record each approve, swap and remove.
Who takes part matters too. In a study of 5,179 customer support agents, published in The Quarterly Journal of Economics, an AI assistant raised issues resolved per hour by 14% on average. Novice and low-skilled agents improved by 34%, while experienced and highly skilled agents saw minimal change. So I'd include merchants who are new to their catalogue and merchants who know it well, and read their results separately.
3. Set a quality bar for AI suggestions
A suggestion only counts if the merchant would publish it. Before the sessions, I'd write down what a usable recommendation looks like for this quiz, such as a product that matches the result's answers.
OpenAI's guide to evaluation best practices is a practical reference here. It recommends defining success criteria first, testing typical cases, edge cases and adversarial cases, and combining metrics with human judgement.
I'd keep acceptance and quality apart. In the consulting study, people using AI on the task outside its capabilities did worse than people working without it. A merchant approving a suggestion is a useful signal, but I'd still check approved products against the quality bar.
Can the merchant publish a quiz faster once we count the time they spend checking and correcting suggestions? How many recommendations would they actually use?
4. Test the fallback when there's no good suggestion
I'd also try a case with too little information or no suitable product in the catalogue. Can the merchant finish setup manually?
I'd set this up on purpose, with one result that has a vague description and one the catalogue can't serve. If the only route to publishing runs through a suggestion, a weak one blocks the merchant. The prototype should let them pick products themselves and publish anyway.
AI suggestions or a product picker: what I'd measure
The thresholds below are just for our own thinking. I'd agree the real ones with the team before the sessions.
1.Time to publish, including review and corrections
Good enough for AI suggestions: the AI version is at least 30% faster than the picker.
I’d choose the product picker if the AI version is no faster once review time is counted.
2.Recommendation quality
Good enough for AI suggestions: at least 7 in 10 suggestions are published unchanged and meet the quality bar.
I’d choose the product picker if merchants swap most suggestions, or approve products that miss the bar.
3.Finishing without a suitable match
Good enough for AI suggestions: every merchant publishes the no-match result without help.
I’d choose the product picker if any merchant gets stuck or gives up when nothing fits.
5. Check the ongoing cost of running it
A prototype can feel impressive in a demo. The cost and effort show up when real people use it repeatedly.
Companies selling AI products have said this publicly. In April 2026, GitHub announced it was moving Copilot to usage-based billing, saying agentic work had driven up inference demand and that it had absorbed most of the rising cost so far. A year earlier, Cursor's CEO explained a pricing change by noting that the hardest requests cost an order of magnitude more than simple ones. Both are products where AI is the core offer. A quiz builder adding one AI step faces the same maths on a smaller scale.
I'd estimate model cost per quiz at expected usage, including retries. Model providers charge per token, so every retry, regenerated suggestion or validation pass is another billed call. I'd check how long merchants wait, how much human review each quiz needs and what it takes to monitor recommendation quality once the feature is live. On top of that sit the engineering build and maintenance.
Setup-time versus per-shopper AI: how the cost changes
For this quiz example, generating suggestions once during merchant setup has different economics from generating them for every shopper.
Generated during setup, the number of model calls follows the number of quizzes and results, plus retries. The merchant waits for suggestions in an admin screen and reviews every product before any shopper sees it. If suggestions can be prepared ahead, for example when a merchant imports their catalogue, OpenAI's Batch API charges 50% less than real-time calls, with results returned within 24 hours.
Generated for every shopper session, the number of calls follows traffic. Every completed quiz triggers a call, so cost rises with each campaign that sends visitors. Response time becomes part of the shopper's experience. Nobody reviews each answer before it appears, so quality checks have to run continuously after launch. Real-time calls also rule out the batch discount.
OpenAI's own latency optimisation guide ends its list of principles with "Don't default to an LLM." If a product picker gives merchants the same result, it has no model bill, no wait and nothing to monitor.
How the success metric changes in B2B and SaaS funnels
In B2B lead qualification where we have an enquiry or booked demo funnel, I'd test whether AI summaries save Sales review time and whether suitable leads still reach a qualified demo. I look at enquiry journeys like this in website strategy and optimisation.
In SaaS onboarding where I measure sign up click to onboarding completion, I'd compare an AI-suggested next step with a predefined onboarding path and measure cost per additional activated user. I work through this kind of trade-off in product growth and monetisation.
AI feature checklist for founders and PMs
Before an AI feature goes on the roadmap, I'd want to tick off each of these:
Watch real users do the task today and note where they get stuck.
Name the simpler alternative, such as a better picker or a predefined path, and test it on the same task and data.
Build a clickable prototype of one step before asking engineering for anything.
Write down the quality bar before the sessions, with typical cases and edge cases.
Time the whole task, including review and corrections, not only the AI step.
Test a case with no good answer and confirm people can still finish.
Include new and experienced users, and read their results separately.
Estimate model cost per task at expected usage, including retries and regenerated output.
Decide whether the AI runs once at setup or on every end-user session, and cost both.
Pick the metric that decides it for your funnel, such as time saved, qualified demos or cost per activated user.
Agree what you'll monitor after launch and how often quality gets checked again.
What I'd hand to engineering and measure after launch
I'd give engineering a decision and a sample they can build from. The handoff would include the prototype, what we learned from users, how the feature should behave UX-wise when it has no useful suggestion, and what we'll measure after launch.
I'd take AI suggestions forward if merchants could publish faster, the recommendations met the quality bar, and the ongoing cost made sense. If a better product picker solved the same problem, I'd ship that.
After launch, I'd measure:
Time from starting a quiz to publishing it.
The share of suggestions merchants publish without a swap.
How often merchants reach the no-match screen, and whether they still publish.
Model cost per published quiz, including retries.
Quality on a regular sample of live recommendations, checked again whenever the prompt or model changes.
Model quality keeps needing attention after launch. I wrote about this from the ML side in lessons from automating customer support contact reasons.
FAQ
How do you know if an AI feature is worth building?
Test a small prototype on the real task. It's worth building when it saves time once review and corrections are counted and meets a quality bar users would accept. People also need to finish when it has no good answer, and it has to cost less to run than the value it adds.
Should product managers build AI prototypes themselves?
I think every product manager working on AI features should be able to put a small, testable version in front of users before handing it to engineering. A clickable prototype of one step is enough to learn whether the idea helps.
What costs should you include for an AI feature?
Include model calls, retries, response time, human review and ongoing quality checks, plus the engineering build and maintenance. Model providers charge per token, so every retry or regenerated answer is another billed call.
Is it cheaper to generate AI output once or for every user?
Generating once during setup usually means fewer calls, and a person can review the output before anyone sees it. Generating for every user session ties cost to traffic and puts response time in front of the customer. Work that can wait may also qualify for batch pricing, which OpenAI offers at 50% less.
What if a simpler option works just as well?
Ship the simpler option. If a better product picker or a predefined path solves the same problem, it has no model bill and nothing extra to monitor.
If you're weighing up an AI feature for your roadmap, I help teams with AI product discovery and prototyping as part of fractional and interim product management.