A/B Testing a Generative AI Feature Whose Outputs Vary from Run to Run

You are the product manager for a feature built on a generative AI model. The model samples its responses, so the same input can produce a different output each time it runs. Design an experiment that tells you whether this feature is working, and explain how your design handles the fact that its outputs are not deterministic. ```hint Two sources of noise Your measurements vary both because users differ and because the model gives a different answer on each run. Work out which parts of your design (the evaluation set, the randomization unit, the sample size, the analysis) each source affects. ``` ### Clarifying Questions - Is the comparison the feature against no feature, or a new model or prompt version against the one already live? - What does the feature produce, such as chat answers, drafts or summaries inside an existing product, or images, and what does a user do with a good output versus a bad one? - Should the test use the exact sampling settings users will get, or can settings or random seeds be fixed during evaluation? - Are there safety or policy checks that must pass before any user is exposed? ### What a Strong Answer Covers - Why output randomness adds variance and makes single-sample judgments unreliable, without making a randomized comparison invalid - A layered plan: offline evaluation with several samples per prompt and calibrated graders, then an online test randomized at the right unit - User-outcome metrics alongside output-quality, consistency, safety, latency and cost measures - Statistical handling: analysis at the user level, a sample size that reflects the extra variance, variance reduction, and enough duration to see past novelty - Grader reliability: agreement between automated and human raters, and controls for position and length bias in side-by-side comparisons ### Follow-up Questions - Your automated grader prefers the new version, but the online test shows no change in user outcomes. Which do you trust, and what do you check first? - How would you test a change whose goal is to make outputs more consistent rather than better on average? - The underlying model was updated partway through your experiment. What do you do with the data you have?