Estimate live sports impact on subscriptions
Overview
This question evaluates a data scientist's competence in causal inference and observational study design within analytics and experimentation, covering skills such as defining treatment and control, selecting outcome metrics and covariates, and choosing identification strategies like matching, IV, and difference-in-differences; it is commonly asked to assess the ability to produce credible causal estimates when randomized experiments are unavailable. It tests both conceptual understanding of identification assumptions (e.g., unconfoundedness, overlap, SUTVA, IV validity) and practical application of matching algorithms, propensity modeling, regression specifications, diagnostics, and staggered-rollout/event-study designs for real-world program evaluation.
Estimate the Causal Impact of Live Sports on Prime Subscriptions and Engagement
Context
Amazon is considering adding live broadcasts of selected sports events to Prime Video. Using observational data, design an analysis to estimate the causal impact on Prime membership subscriptions and user engagement.
Tasks
-
Define treatment and control at the user level
- Specify both an exposure-based definition and an uptake-based definition (e.g., users exposed to or actually watching live sports vs not).
-
Define primary outcome metrics
- Examples: subscription starts, retention, upgrades, watch-time.
- List key covariates to justify selection on observables.
-
Matching design
- Choose a method (k-NN, caliper, Mahalanobis, or propensity-score matching).
- Define the distance metric or the propensity model class.
- Explain overfitting controls (regularization, cross-fitting).
- Specify balance diagnostics you require (SMD thresholds, variance ratios, overlap plots).
-
Assumptions
- State unconfoundedness, overlap, SUTVA, and how you will test or justify them.
-
Instrumental variables
- Propose an instrument based on randomized variation in promotion prominence for the live stream (e.g., hero banner vs standard tile).
- Write the 2SLS explicitly: First stage Z → LiveWatch with controls and fixed effects; second stage LiveWatch_hat → Outcome with the same controls.
- Discuss IV validity: relevance, exclusion, independence, monotonicity; weak-IV checks (first-stage F); and over-identification tests if multiple instruments.
-
DID and event study alternative
- Outline a staggered rollout design.
- Provide the regression specification, fixed effects, heterogeneity, and key threats (interference, spillovers, time-varying confounding) with mitigations.
Solution
Login required