Apply Multiple Testing Corrections for Valid Results Analysis
Company: Attentive
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
A new message-sending feature is A/B-tested separately inside 30 companies (50/50 split, α = 0.
05). Results show 2 significant uplifts, 1 significant decline, and the rest are not significant.
##### Question
What conclusions, if any, can you draw from these results? How should multiple testing be addressed, and which correction methods (e.g., Bonferroni, Benjamini–Hochberg) would you apply before declaring the feature effective?
##### Hints
Compare expected false positives (30 × 0.
05) with observed, discuss family-wise vs. FDR control, and explain implications for rollout decisions.
Quick Answer: This question evaluates competency in statistical inference for experimentation, focusing on multiple testing corrections, interpretation of parallel A/B test outcomes, and the distinction between controlling family-wise error rate and false discovery rate.