Infer causal impact without an A/B test
Company: Google
Role: Data Scientist
Category: Statistics & Math
Difficulty: hard
Interview Round: Technical Screen
Engineering shipped a new version intended to reduce disconnections, but no A/B holdout exists. Rigorously evaluate effectiveness: choose among interrupted time series with seasonality, difference-in-differences using untreated regions/devices, synthetic control from donor pools, or regression discontinuity if rollout timing is sharp; state identification assumptions, pre-trend checks, placebo tests, and robustness to staggered rollout; specify outcome and model family (e.g., binomial for drop rate, Poisson/negative binomial for drop counts), variance estimation (cluster-robust SEs by account/region), and variance reduction via CUPED; define the primary metric (drops per 1k minutes) and guardrails, then compute confidence intervals and minimal detectable change for 80% power at α=0.05 given a baseline of 3.0 drops/1k minutes over 10M daily minutes and realistic autocorrelation; explain how you’ll adjust for confounders such as network mix shifts, seasonality, and changing user composition, and how you’ll communicate uncertainty to stakeholders.
Quick Answer: This question evaluates causal inference and observational study design skills, including time-series and panel methods, identification strategies (e.g., ITS, DiD, synthetic control, RD), statistical modeling and power/sample-size estimation, confounder adjustment, and communication of uncertainty.