Resolve Simpson’s paradox in email A/B test
Company: LinkedIn
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
## Email campaign experiment with Simpson’s paradox
A marketing team tests a new email variant **B** vs control **A**.
The experiment ran for **two weeks** in **two cities** (e.g., SF and NY). When you look **within each city-week segment**, variant **B** appears to outperform **A**. But when you **aggregate all data together**, variant **A** appears to outperform **B**.
### Questions
1. Explain how this situation can occur (**Simpson’s paradox**) in the context of this experiment.
2. How would you determine which email is actually “better” for a product decision?
- What is your **primary metric**?
- What diagnostic/guardrail metrics would you check?
3. What checks would you run to detect issues like **imbalance**, **confounding**, or **time effects**?
4. Can you compute a **confidence interval** (or significance test) for the effect in a way that is robust to the paradox? If yes, how?
5. If you suspect the paradox is caused by experimental flaws (e.g., allocation imbalance), what would you recommend doing next?
Quick Answer: This question evaluates understanding of Simpson's paradox, causal inference, experimental design, metric selection, and statistical inference within A/B testing for email campaigns, testing Analytics & Experimentation competencies for a Data Scientist role.