Design an A/B for rare cancellations

Quick Overview

This question evaluates a data scientist's competency in experimental design, rare-event power analysis, precise metric and guardrail definitions, variance-reduction covariate adjustment, sequential-testing decisions, monitoring/dashboard specification, and interpretation of heterogeneous treatment effects.

Design an A/B for rare cancellations

Company: HBO

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Take-home Project

HBO Max is testing a new autoplay-preview feature on the home page. Weekly cancellations are rare (baseline 0.30% of active subscribers per week). You must design and analyze an online A/B experiment to detect a 0.05 percentage-point absolute change within four weeks. Specify: (1) Unit of randomization and exposure eligibility rules given multi-profile households and shared devices; (2) Primary and guardrail metrics (define exact numerators/denominators and event windows), including weekly cancellation rate, app crashes, and total weekly watch-hours; (3) Power analysis and sample-size calculation under binomial assumptions, with and without CUPED using pre-experiment watch-hours as a covariate; (4) Whether to use fixed-horizon vs. sequential testing (e.g., alpha-spending or group-sequential) and the decision boundary you would implement; (5) A plan for rare-event stability (e.g., stratification, variant-balancing, winsorization for watch-time outliers); (6) A monitoring dashboard spec (in Tableau or similar) showing daily guardrails, cumulative impact, and variant balance; (7) How you will interpret heterogeneous effects by country and device, and how this affects the global ship/no-ship decision.

Quick Answer: This question evaluates a data scientist's competency in experimental design, rare-event power analysis, precise metric and guardrail definitions, variance-reduction covariate adjustment, sequential-testing decisions, monitoring/dashboard specification, and interpretation of heterogeneous treatment effects.

|Home/Analytics & Experimentation/HBO
HBO logo
HBO
Oct 13, 2025, 9:49 PM
hardData ScientistTake-home ProjectAnalytics & Experimentation
1
0

A/B Test Design: Autoplay-Preview on the Home Page

Context: A streaming service is testing an autoplay-preview feature on the home page. Weekly cancellations are rare, with a baseline of 0.30% of active subscribers per week. You must design and analyze an online A/B test to detect an absolute change of 0.05 percentage points in the weekly cancellation rate within a 4-week experiment.

Provide the following:

  1. Unit of randomization and exposure eligibility rules, considering multi-profile households and shared devices.
  2. Primary and guardrail metrics with precise definitions (numerators, denominators, event windows). Include:
    • Weekly cancellation rate (primary)
    • App crashes (guardrail)
    • Total weekly watch-hours (guardrail)
  3. Power analysis and sample-size calculation under binomial assumptions to detect a 0.05 percentage-point change, both:
    • Without CUPED
    • With CUPED using pre-experiment watch-hours as a covariate
  4. Choice of fixed-horizon vs. sequential testing (e.g., alpha-spending or group-sequential) and the exact decision boundary you would implement.
  5. A plan for rare-event stability (e.g., stratification, variant-balancing, winsorization for watch-time outliers).
  6. A monitoring dashboard specification (in Tableau or similar) showing daily guardrails, cumulative impact, and variant balance.
  7. How to interpret heterogeneous effects by country and device, and how this affects the global ship/no-ship decision.
Loading comments...