A/B Test Design: Autoplay-Preview on the Home Page

Context: A streaming service is testing an autoplay-preview feature on the home page. Weekly cancellations are rare, with a baseline of 0.30% of active subscribers per week. You must design and analyze an online A/B test to detect an absolute change of 0.05 percentage points in the weekly cancellation rate within a 4-week experiment.

Provide the following:

1. Unit of randomization and exposure eligibility rules, considering multi-profile households and shared devices.
2. Primary and guardrail metrics with precise definitions (numerators, denominators, event windows). Include:
   • Weekly cancellation rate (primary)
   • App crashes (guardrail)
   • Total weekly watch-hours (guardrail)
3. Power analysis and sample-size calculation under binomial assumptions to detect a 0.05 percentage-point change, both:
   • Without CUPED
   • With CUPED using pre-experiment watch-hours as a covariate
4. Choice of fixed-horizon vs. sequential testing (e.g., alpha-spending or group-sequential) and the exact decision boundary you would implement.
5. A plan for rare-event stability (e.g., stratification, variant-balancing, winsorization for watch-time outliers).
6. A monitoring dashboard specification (in Tableau or similar) showing daily guardrails, cumulative impact, and variant balance.
7. How to interpret heterogeneous effects by country and device, and how this affects the global ship/no-ship decision.