Diagnose and decide on watch-time drop
Overview
This question evaluates a data scientist's skills in causal inference, experiment analytics, metric definition and guardrails, confounder adjustment, and rapid decision-making under time pressure.
Disney
Oct 13, 2025, 9:49 PM
Data Scientist
HR Screen
Analytics & Experimentation
0
0
Hulu iOS Homepage Ranking Model Rollout: Causal Read and Decision Within 48 Hours
Context
- On 2025-08-25, a new homepage ranking model was shipped to 100% of iOS traffic.
- Observation (2025-08-25 to 2025-08-31): average watch-time per session among iOS new users declined by 2.7%; Android was flat.
-
Confounders:
- 2025-08-29: major content premiere.
- 2025-08-27: 20-minute payments outage affecting subscription starts.
- No built-in holdout.
You have 48 hours to recommend: roll back, continue, or mitigate.
Tasks
- Define the primary decision metric(s) and guardrails (e.g., session starts, error rate, churn proxy) and set concrete decision thresholds.
-
Propose an identification strategy without a holdout:
- (a) Difference-in-differences using Android as control with pre-period 2025-08-18–2025-08-24.
- (b) Geo-based synthetic control within iOS.
- State assumptions (parallel trends, no interference), diagnostics, and falsification tests.
- Adjust for seasonality, the 2025-08-29 premiere, and the 2025-08-27 outage. Specify fixed effects/controls. Address heterogeneity by country, platform version, and new vs returning users.
- Quantify impact with confidence intervals and provide a back-of-the-envelope weekly revenue effect (state assumptions for ad- and subscription-revenue paths).
- Outline a fast remediation plan (e.g., revert, throttle, feature flags, guardrail monitors), and a forward plan for a proper September experiment (holdouts, pre-registration, CUPED, exposure logging).
Solution
Login required
Show