Explain Statistical Concepts in A/B Testing and Corrections

Q: Explain Statistical Concepts in A/B Testing and Corrections

This question evaluates understanding of statistical inference in A/B testing—specifically familiarity with p-values, Type I and II errors, statistical power, and multiple-comparison corrections—and the competency to assess experiment validity and control false-positive rates.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Question

A/B Testing: p-values, Power, and Error Rates with Multiple Comparisons

Context

You are reviewing the results of an online A/B experiment. Stakeholders question whether your findings are statistically valid, especially because you track several metrics and may have more than two variants.

Task

Define the following in the context of A/B testing:
- p-value
- Type I error
- Type II error
- Statistical power
Explain why tracking multiple metrics and/or testing multiple variants inflates false positives and requires corrections (e.g., Bonferroni).
Demonstrate with a concrete numerical example how family-wise error rate (FWER) grows with the number of tests and how Bonferroni controls it.

Explain Statistical Concepts in A/B Testing and Corrections

A/B Testing: p-values, Power, and Error Rates with Multiple Comparisons

Context

Task

Solution

Comments (0)