How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

What difficulty level is this interview question?

This is a hard difficulty Statistics & Math question, commonly asked during Technical Screen rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Compute sample size and test duration correctly

Quick Overview

This question evaluates competency in experiment design and statistical power calculations, including sample size estimation, variance misspecification effects, multiple-comparison corrections, sequential monitoring boundaries, and adjustments for covariates or clustering.

Powering Two Online Experiments: Sample Size, Duration, and Design Defenses

You are designing experiments to improve a friend-accept rate metric in a high-traffic consumer app. Assume independent users, a Bernoulli outcome (accept vs not), and that traffic estimates refer to unique eligible users per day.

Baseline accept rate: 9.0% (p0 = 0.09) Eligible traffic: 10M users/day

Scenario A — Two-arm A/B Test

Goal: Detect an absolute lift of +0.5 percentage points (Δ = +0.005) in accept rate.
Test: Two-sided, α = 0.05; power = 0.80; equal allocation (50/50).

Tasks:

Compute the per-arm sample size and the calendar days required.
Recompute both if the true standard deviation is 20% higher than assumed.

Scenario B — Three Arms with Shared Control

Arms: Control (C), Treatment 1 (T1), Treatment 2 (T2)
Allocation: 20% to C, 40% to T1, 40% to T2
Goal: Detect +0.3 pp (Δ = +0.003) for either T1 vs C or T2 vs C with familywise α = 0.05 (two-sided).

Tasks:

Choose and justify Holm vs Bonferroni vs Dunnett for multiple comparisons.
Recompute per-arm sample sizes and total duration under your choice.

For Both Scenarios

Discuss the consequences of daily peeking with a naive p < 0.05 rule; propose a sequential design (e.g., O’Brien–Fleming) and how it changes stopping boundaries.
Explain when you’d switch to user-level CUPED or cluster-robust SEs (e.g., feed-level clustering) and how that affects sample size.

Quick Overview

Powering Two Online Experiments: Sample Size, Duration, and Design Defenses

Baseline accept rate: 9.0% (p0 = 0.09) Eligible traffic: 10M users/day

Scenario A — Two-arm A/B Test

Goal: Detect an absolute lift of +0.5 percentage points (Δ = +0.005) in accept rate.

Test: Two-sided, α = 0.05; power = 0.80; equal allocation (50/50).

Tasks:

Compute the per-arm sample size and the calendar days required.

Recompute both if the true standard deviation is 20% higher than assumed.

Scenario B — Three Arms with Shared Control

Arms: Control (C), Treatment 1 (T1), Treatment 2 (T2)

Allocation: 20% to C, 40% to T1, 40% to T2

Goal: Detect +0.3 pp (Δ = +0.003) for either T1 vs C or T2 vs C with familywise α = 0.05 (two-sided).

Tasks:

Choose and justify Holm vs Bonferroni vs Dunnett for multiple comparisons.

Recompute per-arm sample sizes and total duration under your choice.

For Both Scenarios

Discuss the consequences of daily peeking with a naive p < 0.05 rule; propose a sequential design (e.g., O’Brien–Fleming) and how it changes stopping boundaries.

Explain when you’d switch to user-level CUPED or cluster-robust SEs (e.g., feed-level clustering) and how that affects sample size.

Compute sample size and test duration correctly

Quick Overview

Powering Two Online Experiments: Sample Size, Duration, and Design Defenses

Scenario A — Two-arm A/B Test

Scenario B — Three Arms with Shared Control

For Both Scenarios

Solution

Submit Your Answer to Earn 20XP

Compute sample size and test duration correctly

Quick Overview

Powering Two Online Experiments: Sample Size, Duration, and Design Defenses

Scenario A — Two-arm A/B Test

Scenario B — Three Arms with Shared Control

For Both Scenarios

Solution

Submit Your Answer to Earn 20XP