Measure Super Bowl ad impact with causal design
Company: Tubi
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
Your app ran a 30‑second national Super Bowl ad. Design a measurement plan to estimate incremental impact on installs and revenue. Be specific: (a) Define primary and guardrail KPIs and exact measurement windows (pre‑period, game day by quarter, post‑period). (b) Propose at least two identification strategies given there’s no clean randomized control: e.g., geo‑level difference‑in‑differences across DMAs with heterogeneous ad viewership, synthetic control using similar apps/markets, matched markets, or MMM with a spike regressor. (c) Specify the DiD estimating equation and assumptions (parallel trends, SUTVA, no differential shocks) and how you would test them. (d) Describe how you’ll handle confounders like concurrent promotions, outages, app store featuring, or seasonal traffic spikes during the game and halftime. (e) Show how you would estimate and report uncertainty (CIs via robust or cluster‑robust SEs, placebo tests, randomization inference). (f) Explain how you would validate the result with falsification checks (pre‑trend, ‘placebo Super Bowl’ dates) and heterogeneity (new vs existing users, platforms). (g) If the true lift is a 4% increase in daily installs on game day, outline the data you need and compute the approximate minimum sample (markets and days) to detect it at 5% alpha and 80% power.
Quick Answer: This question evaluates a data scientist's competency in causal inference and experiment design for marketing measurement, covering identification strategies (e.g., DiD and synthetic controls), confounder adjustment, uncertainty quantification, validation checks, and power analysis.