Statistical Inference For Product Analytics

What's being tested

Interviewers probe a candidate’s ability to design, analyse, and interpret randomized experiments in product contexts—translating business questions into a clear estimand, an analysis plan, and diagnostics that protect product health. They want evidence you can pre-specify metrics and guardrails, compute power under realistic clustering/contamination, perform integrity checks (sample-ratio mismatch, balance), and choose robust inference (cluster-robust SEs, permutation/bootstrap, or sequential-testing corrections). At Airbnb this matters because experiments must preserve marketplace health (supply-demand balance, trust metrics) while delivering reliable causal conclusions under operational constraints.

Core knowledge

• Estimand: be explicit whether you target average treatment effect (ATE), intent-to-treat (ITT), or complier average causal effect (CACE); state how noncompliance and exposure logging affect estimands and interpretation.

• Metric design & guardrails: define primary metric (e.g., booking_rate) as a user- or session-level aggregation; pick guardrails (gross_bookings, cancellation_rate, NPS) to catch negative downstream effects and measurement drift.

• Power & sample size (proportions): for two proportions use $n = \frac{(z_{1-\alpha/2}+z_{1-\beta})^2 (p_1(1-p_1)+p_2(1-p_2))}{\Delta^2}$ and plug realistic baseline p and minimum detectable effect Δ.

• Clustering / design effect: compute design effect as $DE = 1 + (m-1)\rho$ where m is cluster size and ICC (intraclass correlation) $\rho$ inflates required n to $n\times DE$; estimate $\rho$ from pilot or historical data.

• Sequential monitoring & false positives: pre-specify stopping rules; use alpha spending (O’Brien–Fleming/Pocock) or always-valid p-values; avoid unadjusted peeking which inflates Type I error.

• Randomization checks: run sample-ratio mismatch binomial/chi-square tests, covariate balance checks, and validate assignment/hash keys in logs to detect routing bugs or instrumentation drift.

• Inference under clustering: prefer cluster-level aggregation with cluster-robust SEs, permutation tests at cluster unit, or mixed-effects models; beware low cluster counts reducing effective DOF.

• Interference & SUTVA: state SUTVA assumptions; for marketplace treatments consider partial interference, network-aware randomization (cluster by host/region) or estimate spillovers explicitly.

• Noncompliance & contamination: default to ITT for unbiased effect on assigned population; use instrumental-variables / CACE (two-stage least squares) when estimating effect on compliers, and report compliance rates.

• Multiple comparisons: pre-specify primary metric; adjust for secondary metrics with Bonferroni or Benjamini–Hochberg when many correlated tests exist; prefer hierarchical testing to control Type I while preserving power.

• Robust CI & small-sample inference: bootstrap or permutation for non-normal metrics; for proportions with small counts use exact (Clopper–Pearson) or Wilson intervals rather than asymptotic normal.

• Data-quality diagnostics: validate exposure-event matching, deduplicate IDs, confirm event timestamps, and check for delayed/late-arriving events before locking analysis.

Worked example — Design an A/B test with causal inference

Start by framing: ask which estimand stakeholders care about (user-level booking rate? gross booking dollars?), the unit of randomization (user, listing, host, city), expected exposure and anticipated contamination sources. Organize your answer into pillars: (1) metric and guardrails; (2) randomization/unit and sample-size (include ICC/design-effect if clustering); (3) integrity monitoring and logging plan; (4) analysis plan (ITT, handling noncompliance, statistical test and corrections). A concrete tradeoff to flag: randomize at the host or city level to avoid interference versus randomizing users for higher power — explain how cluster randomization increases required sample via $DE$. Describe planned diagnostics: sample-ratio tests, pre-period covariate balance, and pre-specified stopping rule (alpha spending) for sequential checks. Close by saying: if I had more time I’d simulate outcome distributions under various contamination/compliance scenarios to validate power and explore sensitivity to ICC and effect heterogeneity.

A second angle — Test conversion difference and adjust for clustering

This problem narrows to hypothesis testing for conversion proportions but adds day- or region-level clustering. Transfer the same concepts: compute per-cluster aggregated conversion rates, estimate ICC from historical day-level variance, inflate sample size by DE, and use cluster-robust SEs or permutation at the cluster level for inference. The constraint changes the analysis choices: with few clusters prefer cluster-aggregated t-tests or randomization inference rather than asymptotic sandwich estimators. Also discuss degrees-of-freedom limits—small numbers of clusters reduce power and require conservative interpretation or alternative designs (stratified randomization, longer experiment window).

Common pitfalls

Pitfall: Ignoring clustering and treating events/users as i.i.d. — leads to anti-conservative SEs and inflated Type I errors; always test for intra-cluster correlation and apply design-effect corrections or cluster-level inference.

Pitfall: Reporting a statistically significant lift without checking business guardrails or downstream metrics — a candidate who only reports p-values misses product risk; always surface guardrail results and practical significance.

Pitfall: Pre-mature peeking with no sequential plan — do not stop on the first positive p-value. Pre-specify an alpha-spending approach or use always-valid methods and report the monitoring plan.

Connections

Interviewers may pivot to heterogeneous treatment effects (causal forests, stratified analysis) or to observational causal tools (difference-in-differences, matching) when randomization is impossible. They may also ask about metric instrumentation and data pipelines as a diagnostics follow-up, but keep answers focused on analysis-level checks and assumptions.

Further reading

• Practical Guide to Controlled Experiments on the Web — Kohavi et al. (Microsoft) — thorough industry patterns for online A/B testing, metrics, and pitfalls.

• [Data Analysis Using Regression and Multilevel/Hierarchical Models — Gelman & Hill] — detailed treatment of clustering, multilevel modeling, and variance decomposition for clustered experiments.