Interpret Results and Address Multiple Testing Concerns

Q: Interpret Results and Address Multiple Testing Concerns

This question evaluates a data scientist's understanding of hypothesis testing, multiple comparisons, false positive control, and the interpretation of A/B experiment outcomes across multiple independent brands.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Experiment Interpretation with Multiple Testing

Context

An A/B experiment is run independently across 30 brands.
Within each brand, users are split 50/50 into Control vs Test.
Per-brand hypothesis tests use α = 0.05 (assume two-sided tests and independence across brands).

Question

You observe:

2 brands show a statistically significant lift,
1 brand shows a statistically significant drop,
27 brands show no statistically significant difference.

What conclusions can you draw from these results? How would you account for multiple testing in your answer?

Interpret Results and Address Multiple Testing Concerns

Experiment Interpretation with Multiple Testing

Context

Question

Solution

Comments (0)