Apply Multiple Testing Corrections for Valid Results Analysis
Overview
This question evaluates competency in statistical inference for experimentation, focusing on multiple testing corrections, interpretation of parallel A/B test outcomes, and the distinction between controlling family-wise error rate and false discovery rate.
Attentive
Aug 4, 2025, 10:55 AM
Data Scientist
Technical Screen
Analytics & Experimentation
56
0
Interpreting 30 Parallel A/B Tests With Mixed Results
Context
- You ran 30 independent within-company A/B tests of a new message-sending feature (50/50 split).
- Per-test significance level: α = 0.05 (assume two-sided unless pre-specified otherwise).
- Outcomes: 2 significant uplifts, 1 significant decline, and 27 non-significant effects.
Tasks
- What conclusions can you draw from these results before and after accounting for multiple testing?
- How should multiple testing be addressed (family-wise error vs. false discovery rate), and which correction methods (e.g., Bonferroni, Holm, Benjamini–Hochberg) would you apply before declaring the feature effective?
- What are the implications for rollout decisions?
Hint: Compare expected false positives (30 × 0.05) with observed, discuss family-wise vs. FDR control, and explain implications for rollout decisions.
Solution
Login required
Show