A/B Testing And Causal Experimentation

What's being tested

These interviews probe a candidate’s ability to design, analyze, and defend causal experiments end-to-end: picking a precise primary metric, sizing and timing an experiment, choosing randomization and analysis methods that respect the unit-of-analysis, diagnosing instrumentation or randomization failures, and translating statistical lift into product recommendations. Google expects Data Scientists to reason about tradeoffs between sensitivity (power), risk (user and revenue impact), and operational constraints while communicating uncertainty clearly.

Core knowledge

• Unit of analysis: Define the experiment at the correct level (user, account, cookie, device). Mismatch causes biased SEs and misleading p-values; cluster randomization when interference or multi-device users exist.
• Primary metric construction: Precisely specify numerator, denominator, unit, aggregation rule, treatment window, and handling of repeated observations (per-user mean, per-event rate, or sum). Example: primary = sum(paid_revenue) per user_id over 28 days.
• Test statistic & SE formulas: For difference in proportions, use $SE=\sqrt{\frac{p_1(1-p_1)}{n_1}+\frac{p_2(1-p_2)}{n_2}}$; for means, $SE=\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}$. Use Welch's t when variances differ.
• Sample size / MDE calc: For two-sided mean test, with equal n: $n=\frac{(Z_{1-\alpha/2}+Z_{1-\beta})^2\cdot 2\sigma^2}{\delta^2}$ where $\delta$ is minimum detectable effect (MDE). Use realistic baseline variance and conversion rates.
• Intent-to-treat (ITT) vs. Treatment-on-treated (TOT): Report ITT as primary; use TOT or instrumental variables to estimate complier effect if noncompliance exists, but note selection bias.
• Covariate adjustment & CUPED: Use pre-period covariates in ANCOVA to reduce variance; CUPED uses a pre-experiment metric to lower $\sigma^2$ and increase power without biasing ITT.
• Multiple comparisons & sequential testing: Control Type I with Bonferroni / BH for many metrics, or use alpha-spending / group-sequential methods (e.g., O’Brien–Fleming) for interim looks; false discovery is common with dashboards.
• Heterogeneous treatment effects: Pre-specify subgroups and interaction tests; beware data-dredging. Use regression with interaction terms for exploratory HTE, and correct for multiplicity when making decisions.
• Duration, ramping, and seasonality: Account for weekdays, marketing events, and user lifecycle. Ensure experiment runs long enough to stabilize metrics (behavioral latency, billing cycles).
• Diagnostics & instrumentation checks: Always run Sample Ratio Mismatch (SRM), unit-consistency tests, pre-period balance, and event-logging completeness. SRM often indicates assignment or logging bugs.
• Non-standard outcomes: For retention or survival outcomes, use Kaplan–Meier and log-rank tests or Cox models for time-to-event; use bootstrap for skewed revenue/LTV.
• Interference & SUTVA violations: If users interact (social features), consider network experiments or cluster/graph-randomization; standard A/B assumptions break otherwise.

Worked example — "Design A/B Test for Subscription Price Increase Effectiveness"

First 30 seconds: clarify the objective (maximize long-run revenue vs. short-term conversion), unit (account or user), and acceptable business constraints (max allowable conversion drop). Pillars of the answer: (1) metric design — primary = 90-day discounted LTV per account_id with secondary metrics conversion_rate and churn; (2) experimental design — randomized rollout at account level, stratify by historical spend or geography; (3) sizing and ramp — compute MDE for revenue per user using baseline mean and variance, plan minimum 8-week run to capture billing cycle; (4) analysis — use ITT for decision-making, regress revenue on treatment with covariate adjustment (ANCOVA), and report bootstrap CI for skewed revenue. Tradeoff to flag: powering for revenue requires large N because variance is high; if business cannot afford long runs, consider powering on conversion (easier) but note mismatch with revenue objective. Close: state guardrails (abort on SRM, large negative churn) and say, "if more time, I'd run a small pilot to validate instrumentation, estimate variance, and consider a price ladder (factorial) to learn elasticity."

A second angle — "Boost Google Workspace Chat Usage with Strategic A/B Testing"

Here the goal is engagement uplift, so metrics and interference look different: primary could be DAU per user or session frequency, and unit is often user_id. Social features create network effects — randomizing at user-level may cause contamination between treated and control (messages cross). You'd consider cluster randomization by team/organization or use exposure modeling. Because engagement changes can be small, use variance reduction (CUPED) and pre-specify subgroup analyses (new users vs. power users). Operationally, real-time metrics and funnel diagnostics matter: if initial increases are from notifications, monitor downstream effects like notification fatigue.

Common pitfalls

Pitfall: Unit-of-analysis mismatch — randomizing on cookie or device while reporting per-user metrics. This underestimates variance and can produce false positives. Always align randomization and analysis unit or use cluster-robust SEs.

Pitfall: Reporting an unanchored metric — saying "lift in conversion = 2%" without stating baseline rate, observation window, or whether it's ITT. Interviewers expect numerator/denominator and CI or p-value.

Pitfall: Ignoring runtime diagnostics and SRM — a statistically significant lift that coincides with a sample-ratio mismatch or logging changes is unreliable. Run SRM, event-completeness, and pre-period balance as part of the analysis pipeline.

Connections

Experimentation work commonly pivots into causal inference on observational data (difference-in-differences, propensity scores) when randomization isn't feasible, or into metric monitoring and anomaly detection to spot experiments that leak. Interviewers may also pivot to ranking/recommender evaluation when experiments affect A/B ranking metrics.

Further reading

• Practical Guide to Controlled Experiments on the Web (Kohavi et al.) — a canonical, practical overview of online A/B testing.
• Evan Miller, A/B Testing: The Complete Guide — hands-on explanations of sample size, sequential testing, and pitfalls.