##### Scenario An A/B test was run in City X and City Y. In each city variant A beats B, yet when combined across both cities variant B looks better. ##### Question How would you interpret these conflicting results? What decision would you make and what additional analysis is needed before rollout? ##### Hints Discuss Simpson’s paradox, weighting by traffic, heterogeneous treatment effects, stratified analysis, and checking sample size.

Evaluates interpretation of conflicting A/B test results across cities using Simpson's paradox and stratified estimands. Strong answers check weighting, SRM, heterogeneity, confidence intervals, and rollout criteria.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at LinkedIn.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at LinkedIn during technical interviews.

Resolve Conflicting A/B Test Results in Cities

A/B Test Paradox Across Two Cities

You ran an A/B test in two geographies, City X and City Y. Within each city, variant A outperforms variant B. However, when the data are pooled across cities, the combined result shows variant B performing better than A.

Interpret the conflicting results, decide what you would do for rollout, and describe additional analysis needed before rollout.

Constraints & Assumptions

Treat city as a possible confounder or stratification variable.
Check whether randomization was balanced within city.
Define the rollout estimand before choosing aggregate or city-level results.
Consider heterogeneous treatment effects and data-quality issues.

Clarifying Questions to Ask

Was randomization stratified by city or global across both cities?
Are baseline outcomes very different between the two cities?
Are traffic weights expected to match future rollout traffic?
Are sample sizes large enough within each city?

Part 1 - Interpretation

How would you interpret the conflicting results?

What This Part Should Cover

Recognize Simpson's paradox or aggregation bias.
Explain how different city traffic weights can reverse the pooled result.
Check SRM, assignment balance, logging, and confidence intervals.
Distinguish statistical contradiction from different estimands.

Part 2 - Rollout Decision

What decision would you make for rollout?

What This Part Should Cover

Avoid launching solely based on the pooled result when city is a confounder.
Prefer stratified or pre-specified weighted estimates aligned with rollout traffic.
Consider city-specific rollout if effects or risks differ.
Require guardrail checks and sufficient power.

Part 3 - Additional Analysis

What additional analysis is needed before rollout?

What This Part Should Cover

Estimate treatment effects by city with confidence intervals.
Use stratified analysis, fixed effects, meta-analysis, or regression with city controls.
Check sample ratio mismatch, covariate balance, experiment exposure, and metric definitions.
Examine whether future traffic mix will resemble the test.

Follow-up Questions

How would you explain this to a product manager?
What if city-level results are positive but underpowered?
How would the decision change if traffic mix will shift after launch?

A/B Test Paradox Across Two Cities

Interpret the conflicting results, decide what you would do for rollout, and describe additional analysis needed before rollout.

Constraints & Assumptions

Treat city as a possible confounder or stratification variable.
Check whether randomization was balanced within city.
Define the rollout estimand before choosing aggregate or city-level results.
Consider heterogeneous treatment effects and data-quality issues.

Clarifying Questions to Ask

Was randomization stratified by city or global across both cities?
Are baseline outcomes very different between the two cities?
Are traffic weights expected to match future rollout traffic?
Are sample sizes large enough within each city?

Part 1 - Interpretation

How would you interpret the conflicting results?

What This Part Should Cover

Recognize Simpson's paradox or aggregation bias.
Explain how different city traffic weights can reverse the pooled result.
Check SRM, assignment balance, logging, and confidence intervals.
Distinguish statistical contradiction from different estimands.

Part 2 - Rollout Decision

What decision would you make for rollout?

What This Part Should Cover

Avoid launching solely based on the pooled result when city is a confounder.
Prefer stratified or pre-specified weighted estimates aligned with rollout traffic.
Consider city-specific rollout if effects or risks differ.
Require guardrail checks and sufficient power.

Part 3 - Additional Analysis

What additional analysis is needed before rollout?

What This Part Should Cover

Estimate treatment effects by city with confidence intervals.
Use stratified analysis, fixed effects, meta-analysis, or regression with city controls.
Check sample ratio mismatch, covariate balance, experiment exposure, and metric definitions.
Examine whether future traffic mix will resemble the test.

Follow-up Questions

How would you explain this to a product manager?
What if city-level results are positive but underpowered?
How would the decision change if traffic mix will shift after launch?

Resolve Conflicting A/B Test Results in Cities

Quick Overview

Resolve Conflicting A/B Test Results in Cities

A/B Test Paradox Across Two Cities

Constraints & Assumptions

Clarifying Questions to Ask

Part 1 - Interpretation

What This Part Should Cover

Part 2 - Rollout Decision

What This Part Should Cover

Part 3 - Additional Analysis

What This Part Should Cover

Follow-up Questions

Write your answer

Resolve Conflicting A/B Test Results in Cities

Quick Overview

Resolve Conflicting A/B Test Results in Cities

A/B Test Paradox Across Two Cities

Constraints & Assumptions

Clarifying Questions to Ask

Part 1 - Interpretation

What This Part Should Cover

Part 2 - Rollout Decision

What This Part Should Cover

Part 3 - Additional Analysis

What This Part Should Cover

Follow-up Questions

Write your answer