A/B Test Design And Sample Size

What's being tested

Interviewers are probing the candidate’s ability to translate a business question into an experiment design with defensible statistical assumptions, compute a correct sample size, and surface operational constraints and diagnostics. They expect a Data Scientist to state clear assumptions (unit of randomization, primary metric, baseline, meaningful uplift), pick the right formula (binary vs continuous vs clustered), and explain tradeoffs like duration vs minimum detectable effect (`MDE`) and adjustments for peeking or multiple comparisons.

Core knowledge

• Definition of MDE: the smallest absolute effect size that matters to the business; sample size scales as $1/\text{MDE}^2$. Always state absolute vs relative uplift and tie to revenue or cost.

• Type I / Type II errors: `alpha` (false positive rate, commonly 0.05) and power ($1-\beta$, commonly 0.8); pick both before computing sample size and report per-arm and total sample.

• Binary metric formula: for equal allocation, per-arm sample for proportions:

  $$n = \frac{\left(Z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} + Z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_1-p_2)^2}$$

  where $\bar p=(p_1+p_2)/2$.

• Continuous metric formula: per-arm sample:

  $$n = 2\left(\frac{Z_{1-\alpha/2}+Z_{1-\beta}}{\Delta/\sigma}\right)^2 = 2(Z_{1-\alpha/2}+Z_{1-\beta})^2\frac{\sigma^2}{\Delta^2}$$

  with $\sigma$ = standard deviation, $\Delta$ absolute difference.

• Cluster / group randomization: multiply by design effect `DE` = 1 + (`m` - 1) `ρ` where `m` = avg cluster size and `ρ` = intra-class correlation; cluster experiments often need orders of magnitude more clusters.

• Allocation ratio and unequal arms: sample size adjusts by factor (1 + 1/`k`) for treatment:control ratio `k`; unequal allocation increases total N for same power.

• Variance reduction techniques: stratification, blocking, and CUPED reduce $\sigma^2$ — directly lowering required N; demonstrate estimating pre-experiment covariance to compute expected reduction.

• Sequential monitoring / peeking: naive peeking inflates Type I error; use alpha-spending (O’Brien–Fleming, Pocock) or sequential tests, or pre-specify a fully Bayesian analysis to avoid corrections.

• Practical traffic/duration tradeoffs: compute required duration = required sample / expected daily eligible users; include seasonality and conversion latency when estimating real run time.

• Pre-experiment checks: confirm randomization via sample ratio mismatch (SRM) (binomial/chi-square test), validate metric instrumentation, and verify logged denominators (exposure counts) in `SQL`.

• Multiple metrics / multiple tests: control family-wise error with Bonferroni or control false discovery rate with Benjamini–Hochberg; report adjusted `p-value`s or pre-register primary metric.

• Realistic scaling numbers: small absolute lifts (0.1% on a 5% baseline) require millions of users; a 1 percentage point lift on a 10% baseline typically needs ~15k per arm (`alpha`=0.05, power=0.8).

Worked example — Calculate Sample Size for Effective A/B Test Design

First 30 seconds: clarify the randomization unit (user, session, cookie), the primary metric (binary conversion vs revenue-per-user), baseline rate or historical mean, the business-meaningful `MDE` (absolute), `alpha` and desired power, allocation ratio, and any clustering or stratification. Skeleton of a strong answer:

1. Estimate baseline and variance from recent telemetry and pick an absolute MDE tied to business (revenue or cost).

2. Choose the appropriate formula (binary vs continuous) and compute per-arm `n`; show the formula and plug numbers.

3. Adjust for clustering (`DE`) or for expected attrition and instrumentation loss by inflating `n`.

4. Plan monitoring: pre-register primary analysis, schedule SRM and instrumentation checks, and decide on alpha-spending if interim looks are needed.

A concrete tradeoff to flag: if traffic cannot support the required `n`, either extend duration, increase `MDE` (accept lower granularity), apply variance reduction (`CUPED`/stratification), or prioritize segments. Close with next steps: run a small pilot to validate variance estimates, and simulate power under plausible ranges; if more time, pre-specify secondary metrics, plan heterogeneity analysis, and draft the `SQL` checks and dashboards.

A second angle

Consider designing sample size when the metric is continuous revenue-per-user and results arrive with long-tailed distributions. Here you'd estimate `σ` either by log-transforming revenue or by Winsorizing extremes; compute `n` on transformed scale and translate back to business units. If randomization is at the store level or experiences interference (customers see multiple treatments), incorporate cluster `DE` and consider hierarchical models for analysis. When multiple treatments or dose levels are tested simultaneously, plan for multiplicity control (FDR or hierarchical testing) and recognize that correcting for many comparisons will meaningfully increase required overall sample or reduce per-comparison power.

Common pitfalls

Pitfall: Reporting a sample size without tying the MDE to business value. Interviewers want you to justify why the effect is worth detecting, not just show algebra.

Pitfall: Using historical average as baseline without checking recency or seasonality — a stale baseline underestimates variance and leads to underpowered tests.

Pitfall: Ignoring clustering or unit-of-analysis mismatch (randomizing users but analyzing sessions) — this inflates Type I error; always align randomization unit with analysis unit or adjust standard errors.

Connections

Interviewers often pivot to metric design (choosing guardrail and secondary metrics) and causal inference topics like SUTVA violations or interference. They may also ask about experiment analysis pipelines (pre-registration, dashboards, SRM checks) or advanced sequential/Bayesian testing strategies.

Further reading

• Practical Guide to Controlled Experiments on the Web — Kohavi et al. — concise industry practices and pitfalls for online A/B testing.

• CUPED: Controlled Experiments with Pre-Experiment Data — Deng et al. — variance reduction technique that often cuts required sample size.