Resolve Conflicting A/B Test Results in Cities

Q: Resolve Conflicting A/B Test Results in Cities

This question evaluates a data scientist's understanding of causal inference, confounding effects such as Simpson's paradox, heterogeneous treatment effects, and the specification of estimands in A/B testing.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

A/B Test Paradox Across Two Cities

Scenario

You ran an A/B test in two geographies (City X and City Y). Within each city, variant A outperforms variant B. However, when you pool the data across both cities, the combined result shows variant B performing better than A.

Questions

How would you interpret these conflicting results?
What decision would you make for rollout?
What additional analysis is needed before rollout?

Hints to Consider

Simpson’s paradox and the role of confounders (e.g., city)
Weighting by traffic and defining the correct estimand
Heterogeneous treatment effects (HTE) across strata
Stratified analysis or meta-analysis approaches
Checking sample size, power, and data quality (e.g., SRM)

Resolve Conflicting A/B Test Results in Cities

A/B Test Paradox Across Two Cities

Scenario

Questions

Hints to Consider

Solution

Comments (0)