Ads Split Test: Diagnosing a 92/8 Traffic Split and an Underpowered Positive Lift

An advertising platform runs a split test on an ad change: the audience is randomly split so that one group (treatment) gets the new version and the other (control) gets the current one, and the two groups are compared on the test's primary metric, such as conversions. The interviewer probes the design and the interpretation in depth: the randomization unit, how units are assigned to treatment and control, pre-treatment balance, statistical power and the minimum detectable effect (MDE), and whether an observed lift is a real effect or an artifact of the setup. The report says that answers such as "run a power analysis" or "check balance" were not accepted on their own: the interviewer kept asking how exactly to check, which metrics to use, and how each problem would change the final business conclusion. Two specific questions anchored the discussion. ### Clarifying Questions - What is randomized: users, devices or cookies, ad accounts, or impressions? Does the 92/8 figure count assigned units, units that were served an ad, or impressions? - What is the primary metric, such as conversions per user, conversion rate or cost per acquisition, and over what attribution window? - Was the 90/10 allocation fixed for the whole test, or was traffic ramped or reconfigured along the way? - Do units enter the analysis when they are assigned, or only once they are served an ad? ### Part 1 — A 90/10 plan that came out 92/8 The test was planned with 90 percent of traffic in treatment and 10 percent in control, but the observed split is 92/8. How do you decide whether this is a problem? Beyond checking for a sample ratio mismatch (SRM), how would you verify that the two groups were balanced before the test started? ```hint Scale changes the verdict Whether 92/8 is alarming depends on how many units were split. Work out what a two-point shift looks like as a test statistic at the size of a typical ads experiment. ``` ```hint When does a unit get counted Ask at what point a unit enters the count. If the treatment changes who gets served an ad, does the count still reflect the random assignment? ``` #### What This Part Should Cover - A formal SRM test and how the sample size changes the verdict - Ads-specific causes of the mismatch and how to localize them - Concrete pre-treatment balance checks with metrics and thresholds, and what a failed check does to the conclusion ### Part 2 — A positive but non-significant lift The result points in the positive direction but is not statistically significant, and you discover that the test was underpowered. How do you interpret and communicate the result? What could have been done before the test started, and what can be done now that it has ended? ```hint Read the interval Look at the range of effects the data are consistent with, not just the p-value. ``` ```hint The split itself costs power Compare how much information a 90/10 split provides with a 50/50 split of the same total traffic. ``` #### What This Part Should Cover - A correct interpretation: inconclusive, with a confidence interval and the effect sizes the test could and could not detect - Pre-test remedies: a business-based MDE, sample sizing, allocation, variance reduction, metric choice and duration - Post-test options and how each changes what the business is told ### What a Strong Answer Covers - Named statistics, metrics and thresholds rather than "check SRM" and "check balance" - Ads-specific failure modes such as exposure-based counting, delivery and budget differences between arms, bot filtering and attribution windows - Quantitative reasoning about power for an unbalanced split - A clear statement of how each finding changes the business conclusion and the recommended next step ### Follow-up Questions - How would you choose the randomization unit for an ads split test, and what goes wrong if you randomize impressions instead of users? - The SRM turns out to come from a single platform. Can you salvage the rest of the test, and how? - How much would CUPED reduce the required sample size, and which pre-period covariate would you use? - The advertiser wants to keep the test running until the result becomes significant. What do you tell them?