Interpret Results and Address Multiple Testing Concerns
Company: Attentive
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
A company runs an A/B experiment on a new message-sending method across 30 independent brands; each brand splits users 50/50 into control and test with α = 0.05.
##### Question
Given that 2 brands show a statistically significant lift, 1 shows a statistically significant drop, and the remaining 27 show no significance, what conclusion(s) can you draw? How would you account for multiple testing in your answer?
##### Hints
Think about expected false positives at α=0.05, family-wise error rate vs. FDR, and whether observed significant results exceed chance.
Quick Answer: This question evaluates a data scientist's understanding of hypothesis testing, multiple comparisons, false positive control, and the interpretation of A/B experiment outcomes across multiple independent brands.