Explain P-value, Confidence Interval, and Multiple Testing Adjustments
Quick Overview
Evaluates A/B testing inference fundamentals, including p-values, confidence intervals, multiple testing adjustments, Type I and Type II errors, z-tests versus t-tests, and CLT versus LLN. Strong answers connect definitions to experiment decisions and common pitfalls.
Explain P-value, Confidence Interval, and Multiple Testing Adjustments
Company: Amazon
Role: Data Scientist
Category: Statistics & Math
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
AB-testing and inferential statistics for new product launches.
##### Question
Define p-value and confidence interval and explain their relationship. How do you adjust for multiple testing (e.g., Bonferroni, Tukey)? Explain Type I and Type II errors with examples. When would you use a Z-test versus a t-test? Compare the Central Limit Theorem with the Law of Large Numbers.
##### Hints
Focus on assumptions, formulas, and practical implications.
Quick Answer: Evaluates A/B testing inference fundamentals, including p-values, confidence intervals, multiple testing adjustments, Type I and Type II errors, z-tests versus t-tests, and CLT versus LLN. Strong answers connect definitions to experiment decisions and common pitfalls.
Explain P-value, Confidence Interval, and Multiple Testing Adjustments
Amazon
Jul 12, 2025, 6:59 PM
mediumData ScientistTechnical ScreenStatistics & Math
45
0
Explain P-Value, Confidence Interval, and Multiple Testing Adjustments
You are running online A/B experiments to evaluate a new product launch. Assume randomized assignment and a binary primary metric such as conversion unless the interviewer states otherwise.
Constraints & Assumptions
Use practical A/B testing examples, not only textbook definitions.
Distinguish statistical significance from practical significance.
Include assumptions behind each test and adjustment method.
Explain common pitfalls clearly.
Clarifying Questions to Ask Guidance
Is the test one-sided or two-sided?
Is the primary metric binary, continuous, count-based, or ratio-based?
How many metrics, variants, and pairwise comparisons are being tested?
Are users independent, or are there clusters or repeated measurements?
Part 1 - P-Value and Confidence Interval
Define the p-value and confidence interval, and explain their relationship.
What This Part Should Cover Guidance
P-value as probability of data at least as extreme under the null.
Confidence interval as a range produced by a procedure with long-run coverage.
Relationship between a two-sided test and whether a confidence interval excludes the null value.
Common misinterpretations.
Part 2 - Multiple Testing Adjustments
How do you adjust for multiple testing? Contrast Bonferroni and Tukey's HSD, and note when you would use each.
What This Part Should Cover Guidance
Family-wise error rate and why multiple comparisons inflate false positives.
Bonferroni as simple and conservative across planned tests.
Tukey's HSD for all pairwise comparisons after ANOVA-style comparisons of group means.
Mention false discovery rate methods when many exploratory metrics are involved.
Part 3 - Type I and Type II Errors
Explain Type I and Type II errors with concrete A/B testing examples.
What This Part Should Cover Guidance
Type I error as launching a feature that has no real lift.
Type II error as missing a real improvement.
Role of alpha, power, sample size, variance, and minimum detectable effect.
Part 4 - Z-Test Versus T-Test
When would you use a Z-test versus a t-test?
What This Part Should Cover Guidance
Z-test for large samples or known variance, common for large-scale binary metrics via normal approximation.
T-test for continuous metrics with unknown variance, especially smaller samples.
Assumptions and robust alternatives.
Part 5 - CLT Versus LLN
Compare the Central Limit Theorem with the Law of Large Numbers and explain practical implications for experiment analysis.
What This Part Should Cover Guidance
LLN as sample averages converging to expected values.
CLT as standardized sample averages becoming approximately normal.
How these justify metric estimation and confidence intervals in large experiments.
What a Strong Answer Covers Guidance
A strong answer gives accurate definitions, links inference concepts to A/B testing decisions, controls false positives across multiple comparisons, and explains when approximations are valid.
Follow-up Questions Guidance
How would you handle many secondary metrics?
What if the p-value is significant but the effect size is tiny?
How would clustering or repeated users change the analysis?
