PracHub
QuestionsPremiumLearningGuidesInterview PrepNEWCoaches
|Home/Analytics & Experimentation/Tubi

Determine A/B test sample size drivers

Last updated: Mar 29, 2026

Quick Overview

This question evaluates statistical power and sample size determination for A/B testing, covering hypothesis testing for proportions, derivation using normal approximations, the impact of MDE, α and power choices, allocation ratios, cluster design (ICC and design effect), group‑sequential monitoring, and adjustment for overdispersed count outcomes.

  • hard
  • Tubi
  • Analytics & Experimentation
  • Data Scientist

Determine A/B test sample size drivers

Company: Tubi

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

For an A/B test with baseline conversion p0=8.0%, you want to detect a relative MDE of 8% (i.e., absolute +0.64 pp) at two‑sided alpha=0.05 and 80% power with 1:1 allocation. (a) Derive and compute the required per‑arm sample size using a normal approximation to the difference in proportions (show the z terms you use). (b) Now, holding everything else constant, state qualitatively and quantitatively how the required n changes when: MDE halves; alpha is tightened to 0.01; power increases to 90%; allocation is 75/25; variance is higher due to user‑level clustering with ICC=0.02 and average cluster size m=5 (compute the design effect); and you adopt group‑sequential monitoring with two equally spaced looks using O’Brien‑Fleming boundaries. (c) If the metric is overdispersed count data (NB2), outline how you would re‑estimate n.

Quick Answer: This question evaluates statistical power and sample size determination for A/B testing, covering hypothesis testing for proportions, derivation using normal approximations, the impact of MDE, α and power choices, allocation ratios, cluster design (ICC and design effect), group‑sequential monitoring, and adjustment for overdispersed count outcomes.

Related Interview Questions

  • Handle multiple experiment metrics and communication - Tubi (hard)
  • Measure Super Bowl ad impact with causal design - Tubi (hard)
Tubi logo
Tubi
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
2
0

A/B Test Sample Size With Variations and Overdispersed Metric Case

You are planning a two‑arm A/B test on a binary conversion metric with:

  • Baseline conversion p0 = 8.0% (0.08)
  • Relative MDE = 8% (so p1 = 1.08 × p0 = 0.0864; absolute lift Δ = 0.0064)
  • Two‑sided α = 0.05, power = 80% (β = 0.20)
  • 1:1 allocation (equal sample sizes per arm)

Assume a z‑test for the difference in proportions using a normal approximation.

Tasks

(a) Derive and compute the required per‑arm sample size using the normal approximation to the difference in proportions. Show the z terms you use.

(b) Holding everything else constant (relative to part a), state qualitatively and quantitatively how the required sample size changes when each of the following is changed individually:

  1. MDE halves.
  2. α is tightened to 0.01 (two‑sided).
  3. Power is increased to 90%.
  4. Allocation is 75/25 (treatment/control).
  5. Observations are clustered at the user level with ICC = 0.02 and average cluster size m = 5 (compute the design effect and apply it).
  6. You adopt group‑sequential monitoring with two equally spaced looks using O’Brien‑Fleming boundaries.

(c) If the primary metric is an overdispersed count (NB2), outline how you would re‑estimate the required sample size.

Solution

Show

Comments (0)

Sign in to leave a comment

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Tubi•More Data Scientist•Tubi Data Scientist•Tubi Analytics & Experimentation•Data Scientist Analytics & Experimentation
PracHub

Master your tech interviews with 7,500+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.