Explain Central Limit Theorem's Importance in A/B Testing
Quick Overview
Evaluates understanding of the Central Limit Theorem and confidence intervals in A/B testing. Strong answers explain sampling distributions, standard error, normal approximations for experiment metrics, caveats for dependence and heavy tails, and compute the 95% confidence interval as [2.902, 3.098].
Explain Central Limit Theorem's Importance in A/B Testing
Company: Meta
Role: Data Scientist
Category: Statistics & Math
Difficulty: easy
Interview Round: Onsite
##### Scenario
General statistics knowledge check during analytics interview.
##### Question
State the Central Limit Theorem and explain why it matters for A/B testing. 2. Given a sample mean of 3, known population standard deviation of 1, and n = 400, construct a 95% confidence interval for the mean. 3. Sketch the sampling distribution on a whiteboard and mark the confidence bounds.
##### Hints
CLT: sampling distribution of the mean approaches normal; CI = mean ± 1.96*σ/√n.
Overview: Evaluates understanding of the Central Limit Theorem and confidence intervals in A/B testing. Strong answers explain sampling distributions, standard error, normal approximations for experiment metrics, caveats for dependence and heavy tails, and compute the 95% confidence interval as [2.902, 3.098].
Explain Central Limit Theorem's Importance in A/B Testing
Meta
Jul 12, 2025
easyData ScientistOnsiteStatistics & Math
18
0
Explain the Central Limit Theorem's Importance in A/B Testing
This statistics prompt asks you to state the Central Limit Theorem, explain why it matters for A/B testing, and construct a confidence interval for a mean with known population standard deviation.
Constraints & Assumptions
Assume independent observations with finite variance.
For the confidence interval, population standard deviation is known.
Use a 95% confidence level and normal critical value.
Distinguish the population distribution from the sampling distribution of the sample mean.
Clarifying Questions to Ask Guidance
Are observations independent, or are users clustered or repeated?
Is the metric a mean, proportion, ratio, or heavy-tailed revenue metric?
Is the population standard deviation known or estimated?
Part 1 - State the CLT
State the Central Limit Theorem and explain why it matters for A/B testing.
What This Part Should Cover Guidance
Sampling distribution of the sample mean becomes approximately normal as sample size grows.
Mean of sampling distribution is the population mean and standard error is
sigma / sqrt(n)
.
Why this enables z-tests and confidence intervals for aggregate metrics.
Practical caveats for dependence, heavy tails, and small samples.
Part 2 - Confidence Interval
Given sample mean 3, known population standard deviation 1, and n = 400, construct a 95% confidence interval for the population mean.
What This Part Should Cover Guidance
Standard error
1 / sqrt(400) = 0.05
.
Margin of error
1.96 * 0.05 = 0.098
.
Confidence interval
[2.902, 3.098]
.
Part 3 - Whiteboard Sketch
Sketch the sampling distribution of the sample mean and mark the 95% confidence bounds.
What This Part Should Cover Guidance
Bell-shaped distribution centered at 3 for the estimate or at the true mean conceptually.
Bounds approximately 1.96 standard errors away.
Clear labeling of lower and upper bounds.
What a Strong Answer Covers Guidance
A strong answer explains the CLT intuitively, connects it to A/B test inference, computes the confidence interval correctly, and notes when CLT approximations may be unreliable.
Follow-up Questions Guidance
How does the CLT apply to conversion rates?
What if user-level revenue is extremely heavy-tailed?