Compute sample size and test duration correctly

Quick Overview

This question evaluates competency in experiment design and statistical power calculations, including sample size estimation, variance misspecification effects, multiple-comparison corrections, sequential monitoring boundaries, and adjustments for covariates or clustering.

Compute sample size and test duration correctly

Company: Meta

Role: Data Scientist

Category: Statistics & Math

Difficulty: hard

Interview Round: Technical Screen

Compute sample size and duration for two experiments and defend your assumptions. Scenario A: Baseline friend-accept rate = 9.0%; target MDE = +0.5 percentage points (absolute); two-sided α=0.05, power=0.80, equal allocation; eligible traffic = 10M users/day. 1) Derive the per-arm sample size and calendar days required; 2) show how results change if the true SD is 20% higher than assumed. Scenario B (three arms, shared control): detect +0.3 pp for either treatment vs control with familywise α=0.05; choose and justify Holm vs Bonferroni vs Dunnett; recompute per-arm n and total duration with 20% traffic to control, 40% each to two treatments. For both scenarios: 3) discuss consequences of daily peeking with a naive p<0.05 rule; propose a sequential design (e.g., O’Brien–Fleming) and how it changes stopping boundaries; 4) explain when you’d switch to user-level CUPED or cluster-robust SEs (e.g., feed-level clustering) and how that affects n.

Quick Answer: This question evaluates competency in experiment design and statistical power calculations, including sample size estimation, variance misspecification effects, multiple-comparison corrections, sequential monitoring boundaries, and adjustments for covariates or clustering.

|Home/Statistics & Math/Meta
Meta logo
Meta
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenStatistics & Math
3
0

Powering Two Online Experiments: Sample Size, Duration, and Design Defenses

You are designing experiments to improve a friend-accept rate metric in a high-traffic consumer app. Assume independent users, a Bernoulli outcome (accept vs not), and that traffic estimates refer to unique eligible users per day.

Baseline accept rate: 9.0% (p0 = 0.09) Eligible traffic: 10M users/day

Scenario A — Two-arm A/B Test

  • Goal: Detect an absolute lift of +0.5 percentage points (Δ = +0.005) in accept rate.
  • Test: Two-sided, α = 0.05; power = 0.80; equal allocation (50/50).

Tasks:

  1. Compute the per-arm sample size and the calendar days required.
  2. Recompute both if the true standard deviation is 20% higher than assumed.

Scenario B — Three Arms with Shared Control

  • Arms: Control (C), Treatment 1 (T1), Treatment 2 (T2)
  • Allocation: 20% to C, 40% to T1, 40% to T2
  • Goal: Detect +0.3 pp (Δ = +0.003) for either T1 vs C or T2 vs C with familywise α = 0.05 (two-sided).

Tasks:

  • Choose and justify Holm vs Bonferroni vs Dunnett for multiple comparisons.
  • Recompute per-arm sample sizes and total duration under your choice.

For Both Scenarios

  1. Discuss the consequences of daily peeking with a naive p < 0.05 rule; propose a sequential design (e.g., O’Brien–Fleming) and how it changes stopping boundaries.
  2. Explain when you’d switch to user-level CUPED or cluster-robust SEs (e.g., feed-level clustering) and how that affects sample size.
Loading comments...