PracHub
QuestionsPremiumCoachesLearningGuidesInterview Prep
|Home/Analytics & Experimentation/HBO

Design an A/B for rare cancellations

Last updated: Mar 29, 2026

Quick Overview

This question evaluates a data scientist's competency in experimental design, rare-event power analysis, precise metric and guardrail definitions, variance-reduction covariate adjustment, sequential-testing decisions, monitoring/dashboard specification, and interpretation of heterogeneous treatment effects.

  • hard
  • HBO
  • Analytics & Experimentation
  • Data Scientist

Design an A/B for rare cancellations

Company: HBO

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Take-home Project

HBO Max is testing a new autoplay-preview feature on the home page. Weekly cancellations are rare (baseline 0.30% of active subscribers per week). You must design and analyze an online A/B experiment to detect a 0.05 percentage-point absolute change within four weeks. Specify: (1) Unit of randomization and exposure eligibility rules given multi-profile households and shared devices; (2) Primary and guardrail metrics (define exact numerators/denominators and event windows), including weekly cancellation rate, app crashes, and total weekly watch-hours; (3) Power analysis and sample-size calculation under binomial assumptions, with and without CUPED using pre-experiment watch-hours as a covariate; (4) Whether to use fixed-horizon vs. sequential testing (e.g., alpha-spending or group-sequential) and the decision boundary you would implement; (5) A plan for rare-event stability (e.g., stratification, variant-balancing, winsorization for watch-time outliers); (6) A monitoring dashboard spec (in Tableau or similar) showing daily guardrails, cumulative impact, and variant balance; (7) How you will interpret heterogeneous effects by country and device, and how this affects the global ship/no-ship decision.

Quick Answer: This question evaluates a data scientist's competency in experimental design, rare-event power analysis, precise metric and guardrail definitions, variance-reduction covariate adjustment, sequential-testing decisions, monitoring/dashboard specification, and interpretation of heterogeneous treatment effects.

HBO logo
HBO
Oct 13, 2025, 9:49 PM
Data Scientist
Take-home Project
Analytics & Experimentation
1
0

A/B Test Design: Autoplay-Preview on the Home Page

Context: A streaming service is testing an autoplay-preview feature on the home page. Weekly cancellations are rare, with a baseline of 0.30% of active subscribers per week. You must design and analyze an online A/B test to detect an absolute change of 0.05 percentage points in the weekly cancellation rate within a 4-week experiment.

Provide the following:

  1. Unit of randomization and exposure eligibility rules, considering multi-profile households and shared devices.
  2. Primary and guardrail metrics with precise definitions (numerators, denominators, event windows). Include:
    • Weekly cancellation rate (primary)
    • App crashes (guardrail)
    • Total weekly watch-hours (guardrail)
  3. Power analysis and sample-size calculation under binomial assumptions to detect a 0.05 percentage-point change, both:
    • Without CUPED
    • With CUPED using pre-experiment watch-hours as a covariate
  4. Choice of fixed-horizon vs. sequential testing (e.g., alpha-spending or group-sequential) and the exact decision boundary you would implement.
  5. A plan for rare-event stability (e.g., stratification, variant-balancing, winsorization for watch-time outliers).
  6. A monitoring dashboard specification (in Tableau or similar) showing daily guardrails, cumulative impact, and variant balance.
  7. How to interpret heterogeneous effects by country and device, and how this affects the global ship/no-ship decision.

Solution

Show

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...

Browse More Questions

More Analytics & Experimentation•More HBO•More Data Scientist•HBO Data Scientist•HBO Analytics & Experimentation•Data Scientist Analytics & Experimentation
PracHub

Master your tech interviews with 8,000+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.