Compute minimum sample size for A/B test
Company: Roblox
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Online Assessment
You are implementing a function to compute the **minimum total sample size** for an A/B test.
You are given:
- `observed`: a 1D array of historical/baseline metric values (continuous outcome) to estimate the outcome standard deviation.
- `alpha`: significance level for a **two-sided** test (e.g., 0.05).
- `power`: desired statistical power (e.g., 0.8).
- `delta`: the minimum detectable absolute change in the mean (treatment mean − control mean) you want to be able to detect.
Assumptions:
- Two-sample **z-test** for difference in means.
- Treatment and control groups are **equal-sized**.
- Outcome variance is the same in both groups and is estimated from `observed`.
- Use the normal approximation (z critical values).
Task:
1) Estimate \(\sigma\) using the sample standard deviation of `observed`.
2) Compute the minimum required per-group sample size \(n\).
3) Return the **minimum total sample size** \(N=2n\) as an integer, **rounded up** to the next integer if needed.
Clearly state the formula you use and any edge-case handling (e.g., `delta <= 0`, `sigma == 0`).
Overview: Evaluates the ability to perform sample size and power calculations for a two-sample z-test—including estimating outcome standard deviation from historical continuous data—and is categorized under Analytics & Experimentation for a Data Scientist role at an implementation-level applied statistics abstraction.
Read the full Roblox Data Scientist interview experience this question came from
Community answers
Answer by positiveman0318
import numpy as np
import scipy.stats import norm
def func(observed, alpha, power, delta):
observed = np.asarray(observed, dtype=float)
mu = np.array(observed).mean()
sigma = np.array(observed).std(ddof=1)
z_alpha = norm.ppf(1-alpha/2)
z_power = norm.ppf(power)
n = (2(sigma2)(z_alpha + z_power)2)/delta**2
n = int(np.ceil(n))
return 2*n