##### Scenario An A/B test was run in City X and City Y. In each city variant A beats B, yet when combined across both cities variant B looks better. ##### Question How would you interpret these conflicting results? What decision would you make and what additional analysis is needed before rollout? ##### Hints Discuss Simpson’s paradox, weighting by traffic, heterogeneous treatment effects, stratified analysis, and checking sample size.

This question evaluates a data scientist's understanding of causal inference, confounding effects such as Simpson's paradox, heterogeneous treatment effects, and the specification of estimands in A/B testing.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at LinkedIn.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at LinkedIn during technical interviews.

Resolve Conflicting A/B Test Results in Cities

A/B Test Paradox Across Two Cities

Scenario

You ran an A/B test in two geographies (City X and City Y). Within each city, variant A outperforms variant B. However, when you pool the data across both cities, the combined result shows variant B performing better than A.

Questions

How would you interpret these conflicting results?
What decision would you make for rollout?
What additional analysis is needed before rollout?

Hints to Consider

Simpson’s paradox and the role of confounders (e.g., city)
Weighting by traffic and defining the correct estimand
Heterogeneous treatment effects (HTE) across strata
Stratified analysis or meta-analysis approaches
Checking sample size, power, and data quality (e.g., SRM)

A/B Test Paradox Across Two Cities

Scenario

Questions

How would you interpret these conflicting results?
What decision would you make for rollout?
What additional analysis is needed before rollout?

Hints to Consider

Simpson’s paradox and the role of confounders (e.g., city)
Weighting by traffic and defining the correct estimand
Heterogeneous treatment effects (HTE) across strata
Stratified analysis or meta-analysis approaches
Checking sample size, power, and data quality (e.g., SRM)

Resolve Conflicting A/B Test Results in Cities

Quick Overview

A/B Test Paradox Across Two Cities

Scenario

Questions

Hints to Consider

Solution

Comments (0)

Resolve Conflicting A/B Test Results in Cities

Quick Overview

A/B Test Paradox Across Two Cities

Scenario

Questions

Hints to Consider

Solution

Comments (0)