Comparing a New Model Against Production: Experiment Design and Running Both in Parallel

A team has trained a new version of a machine learning model and wants to know whether it should replace the model currently serving in production. You are asked to design the evaluation. The report gives this case only in outline: the round was case-based and focused on experimental design, and this question began as how to design an experiment, after which the interviewer asked whether the two models could be tested in parallel. The report does not say what the model does or which metrics matter, so this version leaves those for you to clarify; state your assumptions. ### Clarifying Questions - What does the model do, and what does a user see when its output changes: a ranked list, a yes/no decision, generated text? - Which product or business metric is the model meant to move, and what offline evidence already favors the new model? - How do the two models compare in latency and serving cost, and does production have the capacity to run both on the same requests? - How costly is a bad output: a safety or policy violation, a wrong financial decision, or a mildly worse ranking? ### Part 1 — Design the experiment Design an online experiment to decide whether the new model should replace the current one. Cover the hypothesis, the randomization unit, the metrics, the sample size and duration, and the rule you would use to make the launch decision. ```hint Where the models differ Two models often return the same output for most requests. Think about how many users actually experience a difference, and what that does to the effect you can detect. ``` ```hint Offline is not online Ask what an evaluation on logged data can tell you about the new model, and what it cannot tell you about how users will react to it. ``` #### What This Part Should Cover - A hypothesis and primary metric tied to the product goal, with model-quality, latency and cost guardrails - The randomization unit, consistent assignment, power and duration - A pre-registered decision rule and the main threats to validity ### Part 2 — Test the two models in parallel Can the two models be tested in parallel? Describe the ways of running both models at the same time, what each one can and cannot tell you, and how you would order them before, during and after the experiment from Part 1. ```hint Whose output does the user see? For each way of running both models at once, ask whose output reaches the user. That decides whether user behavior can tell you anything about the new model. ``` ```hint Catching breakage versus measuring lift Some ways of exposing a new model exist to find failures cheaply, not to measure improvement. Sort the options by which of the two they are for. ``` #### What This Part Should Cover - The options for running both models at once and the evidence each produces - Their limits and costs, including the side effects of running a model whose output is not shown - A staged rollout with gates and rollback criteria ### What a Strong Answer Covers - Clarifies the model's task, the product goal and the cost of errors before choosing metrics - Online metrics tied to user or business value, kept separate from offline model metrics and system guardrails - A correctly specified, adequately powered experiment, including how dilution is handled - A clear account of what each way of running two models side by side can and cannot establish - A staged launch plan with explicit decision and rollback criteria ### Follow-up Questions - The new model improves the primary metric but is slower and more expensive to serve. How would you decide whether to launch it? - If the model ranks results, how would you compare the two models by interleaving their results, and why does that need less traffic than an A/B test? What can it not tell you? - Running both models on the same requests shows that they disagree on only a small share of requests. How would you use that before and during the A/B test? - After launch, how would you detect that the new model's gain fades, or that it hurts a segment too small to see in the experiment?