CLT, Confidence Intervals, and Hypothesis Tests
Asked of: Data Scientist
Last updated

-
What it is The central limit theorem (CLT) explains why averages from many independent, similar observations are approximately normal, enabling simple uncertainty calculations. Confidence intervals quantify the plausible range for a population quantity, and hypothesis tests assess whether observed effects are likely under a specified null model.
-
Why interviewers ask about it Product data scientists design and interpret experiments that decide if features (e.g., feed ranking tweaks, notification rules) should ship. You need to size tests, compute intervals around lifts, and explain p-values, error rates, and assumptions so launches are safe and defensible.
-
Core ideas to know
- CLT: sample means are approximately normal under mild conditions; accuracy improves as n grows.
- A 95% confidence interval covers the true value in 95% of repeated, identical studies.
- Hypothesis tests frame null vs. alternative; p-value measures extremeness under the null, not effect probability.
- Type I error (α) and power (1−β) trade off; choose α before peeking at results.
- Confidence intervals and two-sided tests are equivalent for many models: null excluded ↔ significant result.
- Assumptions matter: randomization, independence, stable units; watch for sample ratio mismatch and interference.
- Practical sensitivity: variance reduction (e.g., CUPED), robust/transformations for heavy tails, and pre-specified guardrail metrics.
-
A common pitfall Candidates often say “there’s a 95% chance the true mean is in my 95% interval,” which confuses frequentist coverage with probability about a fixed parameter. Another frequent mistake is peeking—checking significance repeatedly without correction—which inflates false positives and ships bad variants. Many also ignore bucketing or event logging issues that break independence (e.g., sample ratio mismatch), making p-values and intervals meaningless. Strong answers state assumptions, check data quality, and justify the test plan (fixed-horizon or sequential) up front.
-
Further reading
- OpenIntro Statistics — Chapters on CLT, confidence intervals, and hypothesis testing; clear, open textbook with worked examples. https://www.openintro.org/book/os/
- Trustworthy Online Controlled Experiments (Kohavi, Tang, Xu) — Industry-standard guide to designing, running, and interpreting A/B tests at Big Tech scale. https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7A7B518F59
- Statistical Challenges in Online Controlled Experiments: A Review — Survey of pitfalls and modern methods (variance reduction, interference, metrics) used by companies like Meta and Airbnb. https://arxiv.org/abs/2212.11366
Related concepts
- Hypothesis Tests, Confidence Intervals, And P-Values
- Hypothesis Testing And Confidence Intervals
- Statistical Inference, Hypothesis Tests, And Power
- Statistical Inference, Hypothesis Testing, And Power
- Hypothesis Testing, Power, And Confidence Intervals
- Statistical Inference, Power, And Confidence IntervalsStatistics & Math