Design an A/B for rare cancellations
Company: HBO
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Take-home Project
HBO Max is testing a new autoplay-preview feature on the home page. Weekly cancellations are rare (baseline 0.30% of active subscribers per week). You must design and analyze an online A/B experiment to detect a 0.05 percentage-point absolute change within four weeks. Specify: (1) Unit of randomization and exposure eligibility rules given multi-profile households and shared devices; (2) Primary and guardrail metrics (define exact numerators/denominators and event windows), including weekly cancellation rate, app crashes, and total weekly watch-hours; (3) Power analysis and sample-size calculation under binomial assumptions, with and without CUPED using pre-experiment watch-hours as a covariate; (4) Whether to use fixed-horizon vs. sequential testing (e.g., alpha-spending or group-sequential) and the decision boundary you would implement; (5) A plan for rare-event stability (e.g., stratification, variant-balancing, winsorization for watch-time outliers); (6) A monitoring dashboard spec (in Tableau or similar) showing daily guardrails, cumulative impact, and variant balance; (7) How you will interpret heterogeneous effects by country and device, and how this affects the global ship/no-ship decision.
Quick Answer: This question evaluates a data scientist's competency in experimental design, rare-event power analysis, precise metric and guardrail definitions, variance-reduction covariate adjustment, sequential-testing decisions, monitoring/dashboard specification, and interpretation of heterogeneous treatment effects.