A/B testing a search recommendation change: power, peeking, mixed metrics, novelty

A search product is preparing to launch an adjustment to its search recommendations. You are asked how you would design an experiment to test it, and the interviewer then probes power, early stopping, conflicting results and novelty, one layer at a time. ### Clarifying Questions - What exactly changes: the suggestions shown while a query is typed, the related searches shown with results, or the recommended results themselves? - What is the change meant to improve: how often searches succeed, how quickly users find what they want, overall engagement, or revenue? - Are most searchers logged in, so that a user's assignment can stay consistent across devices? - Does the recommendation model retrain during the test on interactions logged from both groups? - How much eligible search traffic is there per day, and is there a deadline for the launch decision? ### Part 1 — Design the experiment How would you design the experiment? Choose the randomization unit, define the primary metric and the guardrail metrics, and say whether you would run an A/A test before launching it. ```hint Who sees which version If the same user could get the new recommendations on one search and the old ones on the next, ask what happens to user-level metrics and to what the user learns. ``` ```hint A click is not a success A recommendation that attracts more clicks can still leave users searching again. Think about what a successful search looks like in the logs. ``` #### What This Part Should Cover - The randomization unit, the analysis unit, and how a mismatch between them affects variance - A primary metric tied to the change's goal, with diagnostic and guardrail metrics - What an A/A test would verify and when it is worth running ### Part 2 — Estimate power and sample size How would you estimate the sample size the experiment needs? If the baseline value of the primary metric changed, how would the required sample size change? ```hint Track the variance For a rate metric, the variance depends on the baseline rate itself. Check whether the minimum detectable effect is stated as an absolute change or a relative one. ``` #### What This Part Should Cover - The inputs to a power calculation and the sample-size formula - How the metric's variance is estimated, including for ratio metrics - The direction and size of the change in required sample size when the baseline moves, for absolute and relative effect sizes - Turning a sample size into a test duration ### Part 3 — Stop early and peek Can the experiment be stopped before its planned end? Say when stopping early is justified and when it is not. Is it acceptable to look at the results while the test is running, and if so, how should that be handled? ```hint Every look is a chance to be fooled With no true effect, a running p-value still moves up and down. Ask how often it dips below the threshold at least once if you check it every day. ``` #### What This Part Should Cover - Legitimate reasons to stop: harm, a broken experiment, or a pre-planned boundary - Why repeated looks inflate the false-positive rate, and roughly by how much - Designs that allow monitoring without that inflation, and what they cost ### Part 4 — Interpret mixed results The results come back with some metrics up and others down. How do you decide what to do? ```hint Look for one story Ask whether the movements fit a single explanation of how users' behavior changed, or whether some of them could be noise across many metrics. ``` #### What This Part Should Cover - Checking validity and statistical noise before interpreting, including multiple comparisons - Finding a mechanism that explains the pattern - A decision rule based on the pre-registered primary metric, the guardrails and explicit trade-offs ### Part 5 — Detect and handle a novelty effect How would you tell whether the measured effect is a novelty effect, and what would you do about it? ```hint Effect over exposure time Compare the effect for users in their first days of seeing the change with the effect for the same kind of users later on. ``` #### What This Part Should Cover - Diagnostics based on how the effect changes with time since first exposure - Cohort and new-user comparisons, and the biases they can introduce - How the test length and the launch decision account for novelty ### What a Strong Answer Covers - A design in which the randomization unit, the analysis unit and the metrics are consistent with each other - Correct power reasoning with a formula or numbers, and the correct direction of the baseline effect - A stopping policy that separates harm monitoring from decisions about the effect, with a valid way to look early - Decisions driven by a pre-registered primary metric, awareness of multiple testing, and an explicit trade-off rule - Awareness of time dynamics: novelty, change aversion and weekly cycles ### Follow-up Questions - If the recommendation model retrains on interactions logged from both groups, how could that bias the comparison, and how would you prevent it? - Clicks per search is a ratio of two per-user counts when users are randomized. How do you compute its variance? - How would CUPED change the required sample size, and which pre-period covariate would you use? - How would you choose the minimum detectable effect in the first place?