Online Experimentation And A/B Testing

What's being tested

This topic tests whether you can evaluate an ML model in production rather than relying only on offline validation. You should connect randomized experimentation, model-quality metrics, guardrails, statistical uncertainty, and operational signals such as latency and error rate. Microsoft cares because an MLE must safely determine whether a deployed model improves user-facing outcomes while preserving reliability, fairness, and online/offline consistency.

Core knowledge

• Randomized controlled experiments assign eligible units—typically users, tenants, sessions, or queries—to treatment and control using a stable hash. Randomization prevents systematic differences, but the unit must match the interference pattern: user-level assignment is safer than request-level assignment when users can receive mixed behavior.

• Treatment effect is usually estimated as $\hat{\tau}=\bar{Y}_T-\bar{Y}_C$, with a confidence interval derived from the variance of the two groups. For a ratio metric such as clicks per impression, use a principled delta-method or bootstrap approach rather than independently testing numerator and denominator.

• Primary metrics should measure the model behavior being changed: `NDCG@10`, `MRR`, recall, calibration error, or conversion rate. Define one primary success criterion before launch; otherwise repeatedly inspecting many metrics increases false-positive risk through multiple comparisons.

• Guardrail metrics detect regressions that a quality metric can hide: `p95`, `p99`, latency, timeout rate, empty-result rate, crash rate, CPU utilization, revenue-protection constraints, and complaint or abuse rates. A model that improves ranking while violating latency or safety thresholds should not ship.

• Statistical power depends on effect size, variance, traffic, and significance threshold. A rough continuous-metric approximation is $n \propto (z_{1-\alpha/2}+z_{1-\beta})^2\sigma^2/\delta^2$ per arm; detecting half the effect generally requires roughly four times the sample size.

• Sample-ratio mismatch (SRM) occurs when observed treatment/control allocation differs materially from the intended split. Investigate assignment logic, eligibility filters, exposure logging, and duplicate identities before interpreting outcomes; an SRM can invalidate otherwise impressive results.

• Exposure and intent-to-treat must be distinguished. Analyze users according to their randomized assignment for the primary estimate, even if serving fails for some requests; conditioning only on successfully exposed users can introduce post-treatment selection bias.

• Interference and contamination violate the usual independence assumption. Shared caches, collaborative recommendations, marketplace supply, or models retrained using treatment-generated data can let one arm affect another; use cluster, tenant, or geo-level randomization when individual assignment is unsafe.

• Sequential monitoring changes the false-positive rate if a team repeatedly stops when a result looks significant. Predeclare a fixed duration, use an alpha-spending or sequential-testing method, or report confidence bounds under a valid sequential framework rather than treating every peek as independent.

• Offline/online parity requires identical feature definitions, transformations, missing-value handling, model versions, and label windows. Compare feature distributions and prediction outputs between training, shadow, and production traffic; a high offline `AUC` does not compensate for serving-time feature skew.

• Heterogeneous treatment effects can reveal regressions hidden by an aggregate average: new versus returning users, language, device, geography, query class, or traffic intensity. Predefine important slices, control multiplicity, and require adequate sample sizes; post-hoc slice hunting is hypothesis generation, not proof.

• Experiment safety benefits from shadow mode, canary rollout, ramp limits, automatic rollback, and immutable model/version metadata. Keep the assignment stable across requests and preserve enough exposure, prediction, feature, and outcome information to reproduce the analysis without rebuilding the serving path.

Worked example

Evaluating a New Ranking Model with an Online A/B Test is a representative MLE scenario. In the first 30 seconds, clarify the randomization unit, eligibility rules, exposure definition, primary ranking metric, minimum detectable improvement, experiment duration, and rollback thresholds. I would organize the answer around four pillars: deterministic assignment, valid exposure and outcome logging, offline/online metric parity, and statistical analysis with operational guardrails. I would assign users to the old or new model using a stable hash, then compare `NDCG@10` or downstream qualified engagement while tracking `p99` latency, empty results, and serving failures. The main tradeoff is user-level assignment, which prevents cross-request inconsistency, versus query-level assignment, which may provide cleaner per-query comparisons but contaminates users with mixed experiences. Before trusting the result, I would check SRM, missing exposure events, feature skew, delayed labels, and whether retraining caused treatment data to leak into control. I would ship only if the primary metric clears the predeclared threshold and all guardrails remain within bounds. If I had more time, I would add preplanned segment analysis, a longer-horizon retention check, and a rollback drill.

A second angle

A related scenario is Canarying a New Feature Pipeline Before Full Experimentation, where the central risk is not only model quality but serving correctness. The model can run in shadow mode against mirrored requests, allowing comparison of predictions, feature freshness, null rates, and latency without exposing users to treatment behavior. The framing shifts from estimating causal lift to detecting parity and reliability failures before randomization. Once the canary is healthy, the same stable assignment and guardrail framework supports a causal A/B test.

Common pitfalls

Pitfall: Treating offline `AUC` or validation loss as proof of online improvement. Offline labels may be biased, stale, or disconnected from the serving objective; connect the metric to the deployed decision and validate it experimentally.

Pitfall: Saying “run the test until the p-value is below 0.05.” This ignores power, sequential peeking, delayed outcomes, SRM, and multiple metrics; state the stopping rule, minimum detectable effect, and guardrails before launch.

Pitfall: Focusing on statistical significance while ignoring MLE failure modes. A strong answer should explicitly cover model/version traceability, feature skew, exposure logging, latency, rollback, and what happens when the new model times out.

Connections

The interviewer may pivot to model monitoring and drift detection, feature-store and training-serving skew, canary deployment, counterfactual evaluation, or fairness across slices. Be ready to explain how experiment results feed deployment gates and how delayed or biased labels affect monitoring.

Further reading

• “Controlled Experiments on the Web: Survey and Practical Guide” — Ronny Kohavi et al.’s practical treatment of online experimentation, metrics, and common validity failures.

• “Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing” — detailed coverage of experiment design, SRM, power, guardrails, and sequential analysis.

• “The ML Test Score” — Google’s framework for testing production ML systems beyond model accuracy.