Estimate live sports impact on subscriptions
Company: Amazon
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
Amazon is considering adding live broadcasts of selected sports events to Prime Video. Using observational data, estimate the causal impact on Prime membership subscriptions and engagement. Precisely specify: (1) treatment and control at the user level (e.g., users exposed to/actually watching live sports vs not), (2) primary outcomes (e.g., subscription starts, retention, upgrades, watch‑time), and (3) key covariates for selection on observables. Start with matching: choose a method (k‑NN, caliper, Mahalanobis, or propensity‑score matching), define distance or the propensity model class, explain overfitting controls (regularization, cross‑fitting), and provide balance diagnostics you will require (SMD thresholds, variance ratios, overlap plots). State assumptions (unconfoundedness, overlap, SUTVA) and how you will test/justify them. Then propose an instrument to address unobservables via randomized variation in promotion prominence for the live stream (e.g., hero banner vs standard tile). Write the 2SLS explicitly: First stage Z→LiveWatch with controls and fixed effects; second stage LiveWatch_hat→Outcome with the same controls. Discuss IV validity (relevance, exclusion, independence, monotonicity), weak‑IV checks (first‑stage F), and over‑identification tests if multiple instruments. Finally, outline a DID/event‑study alternative using staggered rollout, detail the regression, fixed effects, and heterogeneity, and list key threats (interference, spillovers, time‑varying confounding) with mitigation.
Quick Answer: This question evaluates a data scientist's competence in causal inference and observational study design within analytics and experimentation, covering skills such as defining treatment and control, selecting outcome metrics and covariates, and choosing identification strategies like matching, IV, and difference-in-differences; it is commonly asked to assess the ability to produce credible causal estimates when randomized experiments are unavailable. It tests both conceptual understanding of identification assumptions (e.g., unconfoundedness, overlap, SUTVA, IV validity) and practical application of matching algorithms, propensity modeling, regression specifications, diagnostics, and staggered-rollout/event-study designs for real-world program evaluation.
