A/B Testing

What's being tested

The interviewer is probing your ability to design and analyze rigorous randomized experiments that support causal claims in a marketplace. Expect to demonstrate clear estimand formulation, principled choice of randomization unit, power/sample-size calculation (including clustering), handling noncompliance and contamination, and an analysis plan that includes quality checks and variance reduction. Airbnb cares because experimental decisions affect hosts and guests asymmetrically and can produce platform-level externalities; the interviewer wants to see statistical rigor plus product-aware tradeoffs.

Core knowledge

• Estimand: clearly state the causal quantity (e.g., Intent-to-Treat (ITT) lift, Complier Average Causal Effect (CACE)), and how it maps to assignment vs. exposure signals in logs.

• Randomization unit: choose between user-level, listing-level, host-level, or cluster-randomization by weighing contamination risk, statistical power, and operational feasibility.

• Sample-size / power: for a difference in means, use $n=\frac{(Z_{1-\alpha/2}+Z_{1-\beta})^2 2\sigma^2}{\delta^2}$ and multiply by design effect $DE=1+(m-1)\rho$ for clustered designs.

• Clustering & ICC: estimate intraclass correlation (ICC) from historical data; even small $\rho$ inflates required sample when cluster size $m$ is large.

• Noncompliance & IV: analyze assignment-based ITT as primary; use instrumental variable (IV) to estimate CACE: $CACE=\frac{\text{ITT}}{\text{compliance rate}}$ under monotonicity and exclusion assumptions.

• Contamination & interference: when SUTVA fails, consider exposure mapping, partial interference assumptions, or cluster randomization; anticipate bias toward the null from spillovers.

• Sequential testing: pre-specify look schedule and use alpha-spending (O’Brien–Fleming / Pocock) or adjust via group-sequential methods; otherwise control Type I inflation.

• Variance reduction: use covariate adjustment (ANCOVA) or CUPED with pre-experiment covariates to increase power without inflating Type I error.

• Metrics & guardrails: pick one primary metric (business-significant, measurable), plus diagnostic and safety/guardrail metrics (e.g., booking_rate, latency, refund_rate) and define exact aggregations (user-level vs. session-level).

• Integrity checks: log and monitor Sample Ratio Mismatch (SRM), assignment fidelity, treatment exposure rates, and delayed-event completeness; run A/A tests to validate pipeline.

• Multiple comparisons: when testing many segments or metrics, control false discovery (Bonferroni, Benjamini–Hochberg) or pre-specify hierarchical testing to preserve power.

• Practical data signals: derive assignment from reliable sources (treatment_assignment table or randomization seed), and define exposure windows and eligibility consistently to avoid immortal-time bias.

Tip: always declare the primary estimand, the unit of analysis, and the stopping rule within the first minute of your plan.

Worked example — Design and Analyze Airbnb Locker Experiment

Start by clarifying scope: who is eligible (guests, hosts, or listings), operational constraints (number of lockers, geographies), and the timescale for bookings vs. locker use. Organize the answer around 4 pillars: (1) estimand (e.g., effect on booking_rate per exposed guest — ITT), (2) randomization (likely listing- or neighborhood-cluster randomization to avoid contamination between co-located guests), (3) metrics, power, and sample size (compute detectable lift for expected baseline booking rate, inflate by design effect), and (4) analysis & diagnostics (ITT primary, per-protocol secondary; SRM checks; covariate adjustment using historical booking propensity). A key tradeoff: listing-level assignment reduces contamination but increases ICC and sample requirements; user-level assignment has more power but higher spillover risk if guests see lockers tied to listings. Close by proposing guardrails (cancellation_rate, host_response_time) and by saying "if I had more time I’d precompute ICC from historical neighborhood cohorts, run an A/A, and simulate power under several contamination scenarios."

A second angle — Design a network-aware Wi‑Fi badge experiment

This problem foregrounds interference: badges on the same network create spillovers. Frame the estimand as an exposure-conditional causal effect (e.g., effect on engagement for users whose router has >=1 treated badge). Consider partial interference by treating networks (or access points) as independent clusters and randomizing at the network-level. Use exposure mapping to define treatment levels (0,1,≥2 badges) and estimate dose–response or pairwise contrasts. Analysis methods include cluster-randomized inference, permutation tests respecting network blocks, and regression with network-size controls. Explicitly discuss how increasing cluster size inflates sample needs via the design effect and how to detect violations by checking cross-network traffic or attribute overlap.

Common pitfalls

Pitfall: Ignoring clustering — designers often compute sample size as if observations are independent; forgetting the ICC leads to underpowered experiments and misleadingly small p-values. Always estimate and apply the design effect.

Pitfall: Vague estimand and unit of analysis — saying “improve bookings” without specifying ITT vs. treated-exposed, or user- vs. listing-level aggregation, makes results uninterpretable; state aggregations and denominators precisely.

Pitfall: Uncontrolled peeking and multiple looks — analysts run mid-test checks without pre-specified stopping rules; this inflates Type I error. Either lock the look schedule or use alpha-spending/group-sequential methods and record all interim analyses.

Connections

Interviewers may pivot to observational causal inference (propensity scores, DiD) when experiments are infeasible, or to uplift modeling for heterogeneous treatment effect estimation. They might also ask about metric design for recommender evaluation or ML model evaluation metrics (AUC, calibration) as extensions.

Further reading

• Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — Kohavi et al., Microsoft-style handbook with practical checks and examples.

• Imbens, G., & Rubin, D., "Causal Inference for Statistics, Social, and Biomedical Sciences" — rigorous treatment of IV, compliance, and potential outcomes.