Experiment Metric Design

What's being tested

Interviewers are checking whether you can design robust experiment metrics and an analysis plan that yield a credible causal conclusion for product changes. Expect to show judgment on unit of randomization, precise primary/guardrail metric definitions, statistical power/sample-size, handling interference/spillovers, and a defensible analysis (ITT, adjustments, sequential monitoring). Airbnb cares because small metric mis-specifications or ignored interference in a marketplace can produce dangerously wrong launch decisions.

Core knowledge

• Primary metric: define a single business-facing numerator and denominator clearly (e.g., bookings per eligible session). Use per-user aggregation when treatment is user-targeted to avoid inflated significance from repeated sessions.

• Guardrail metrics: at least 2–3 safety metrics (e.g., cancellations, time-to-book, host response rate). Guardrails catch negative side effects that a single primary metric would miss.

• Unit of randomization & interference: choose between listing-level, host-level, user-level, or cluster-randomization. If SUTVA is violated, prefer cluster or network-aware randomization and report loss of statistical power.

• Sample size / MDE: for two-sample means/proportions per-arm sample size
  $n = \frac{(Z_{1-\alpha/2}+Z_{1-\beta})^2(\sigma_1^2+\sigma_2^2)}{\Delta^2}$
  For proportions, replace variance by $p(1-p)$. Always convert MDE to absolute units tied to business.

• Power vs variance reduction: use blocking/stratification on strong covariates (region, device) or pre-experiment covariate adjustment (ANCOVA) to reduce $\sigma^2$ and lower required n.

• Analysis estimand: report Intention-to-Treat (ITT) as primary; include as-treated or complier average causal effect (CACE) only with careful compliance measurement.

• Sequential monitoring: pre-specify stopping rules; use alpha-spending, O'Brien–Fleming, or a fixed group-sequential design. Unadjusted peeking inflates Type I error.

• Multiple comparisons: control family-wise error with Bonferroni for a few hypothesis tests or Benjamini–Hochberg for many correlated exploratory metrics.

• Diagnostics & QA: run a sample ratio test (SRT) daily, A/A checks, and pre/post balance on key covariates; flag metric leakage and unexpected denominator changes.

• Aggregation level and variance estimation: cluster-robust SEs if randomization is clustered; use bootstrapping or permutation tests when metric distributions are heavy-tailed (e.g., booking value).

• Conversion vs value metrics: for sparse but high-variance revenue, prefer revenue-per-user aggregated at user-level with winsorization or transform (log1p) in robustness checks.

• Heterogeneous treatment effects: pre-specify a few subgroup analyses (country, guest vs host) and correct for multiplicity; treat exploratory HTEs as hypothesis-generating.

Worked example — Design and Analyze Airbnb Locker Experiment

First 30 seconds: confirm objective (increase bookings, convenience, or host adoption?), eligible populations (guests in city X, hosts with listings near lockers?), and treatment variants (show lockers on search, require opt-in, or incentivize hosts?). Pillars to organize your answer: (1) unit of randomization — likely listing or neighborhood-cluster because lockers are location-bound and create interference across guests; (2) metrics — primary: bookings-per-eligible-user within 14 days; guardrails: cancellations, host occupancy rate, search CTR; (3) power & sample size — compute MDE on booking rate using baseline p and variance, account for ICC if clustered; (4) analysis plan — ITT estimates with cluster-robust SEs, pre-specified covariate adjustment, and sequential monitoring rules. Key tradeoff: clustering by neighborhood reduces interference but inflates required sample size due to intracluster correlation — explicitly quantify expected ICC and show how it increases n. Close by proposing an A/A run or pilot and stating: "If I had more time, I'd instrument locker-views and time-to-book for mediation analysis and pre-register subgroup analyses by traveler intent."

A second angle — Design a network-aware Wi‑Fi badge experiment

This framing forces interference to the foreground: users share a Wi‑Fi network so treatment leaks across connected nodes. Use graph-cluster randomization (cluster users by access-point or community) or design an encouragement design where only some nodes receive prompts and network effects are modeled. Analysis should use randomization-based inference (permutation tests respecting cluster structure) or GEE with robust SEs; estimate direct and spillover effects separately if possible. Power suffers drastically with dense networks, so pilot to estimate effective sample size and report network degree distribution. Pre-specify any allowed cross-cluster interactions and the estimand (total effect vs direct effect).

Common pitfalls

Pitfall: Aggregating at the wrong unit (e.g., session-level when treatment is assigned per-user) yields underestimated SEs and inflated significance.
Always align the analysis aggregation level with the randomization unit; use cluster-robust SEs where applicable.

Pitfall: Hand-wavy metric definitions — saying "increase bookings" without defining numerator, denominator, attribution window, and exclusion rules leads to untestable claims.
Write metric specs like API contracts: numerator, denominator, attribution logic, lookback/forward windows, and failure cases.

Pitfall: Ignoring sequential peeking and multiple metrics — running daily unadjusted tests or chasing p<0.05 on many metrics will produce false positives.
Pre-register stopping rules and use appropriate multiplicity corrections; treat post-hoc discoveries as exploratory.

Connections

Expect pivots toward heterogeneous treatment effects (how to discover and validate subgroups), causal inference with interference (partial interference, spillover estimands), or metric instrumentation & monitoring (how to detect data leakage and late-arriving events). Be prepared to discuss how experiment results feed product decision thresholds and risk tolerances.

Further reading

• Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained (Kohavi et al., KDD 2012) — practical lessons from large-scale experiments.

• Sample Size Calculations and A/B Testing Practical Guide (Evan Miller) — concise formulas and intuition for MDE and sample size.