Detect and address Simpson’s paradox

Q: Detect and address Simpson’s paradox

This question evaluates understanding of aggregation bias and Simpson's paradox, proficiency with stratified or mixed-model estimators, heterogeneity testing, and experiment decisioning within Analytics & Experimentation for a Data Scientist role.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Experiment Aggregation Bias and Heterogeneity: Simpson's Paradox, Robust Estimation, and Decisioning

Context

You ran a randomized experiment measuring conversion rate uplift. The pooled (aggregate) analysis shows +1.2 percentage points (pp) uplift. When stratifying by user segment, the treatment effects are −0.5pp for New users and +2.0pp for Returning users.

Your goal:

Explain how differences in segment mix across arms can produce aggregation bias (often referred to as Simpson’s paradox in this context).
Pre-register an analysis plan that protects against this bias.
Specify how to test for treatment heterogeneity and how that translates to a rollout decision.

Tasks

(a) Construct a concrete numeric example where the overall uplift is +1.2pp even though the segment-level effects are −0.5pp (New) and +2.0pp (Returning), due to a shift in segment mix.

(b) Pre-register a stratified estimator (or MMRM alternative) to estimate a population-average effect that is robust to aggregation bias.

(c) Specify an interaction test for heterogeneity and a decision rule (segment-specific rollout vs. global ship) when heterogeneity is material.

Detect and address Simpson’s paradox

Quick Overview

Detect and address Simpson’s paradox

Experiment Aggregation Bias and Heterogeneity: Simpson's Paradox, Robust Estimation, and Decisioning

Context

Tasks

Write your answer