An experiment shows +1.2pp overall conversion uplift, yet the segment‑level effects are −0.5pp (New) and +2.0pp (Returning). a) Construct a concrete example showing how a shift in segment mix can create Simpson’s paradox. b) Pre‑register a stratified estimator (or MMRM) that protects against aggregation bias. c) Specify an interaction test for heterogeneity and a decision rule when heterogeneity is material (e.g., segment‑specific rollout vs. global ship).

This question evaluates understanding of aggregation bias and Simpson's paradox, proficiency with stratified or mixed-model estimators, heterogeneity testing, and experiment decisioning within Analytics & Experimentation for a Data Scientist role.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Detect and address Simpson’s paradox | Meta Interview Question

Experiment Aggregation Bias and Heterogeneity: Simpson's Paradox, Robust Estimation, and Decisioning

Context

You ran a randomized experiment measuring conversion rate uplift. The pooled (aggregate) analysis shows +1.2 percentage points (pp) uplift. When stratifying by user segment, the treatment effects are −0.5pp for New users and +2.0pp for Returning users.

Your goal:

Explain how differences in segment mix across arms can produce aggregation bias (often referred to as Simpson’s paradox in this context).
Pre-register an analysis plan that protects against this bias.
Specify how to test for treatment heterogeneity and how that translates to a rollout decision.

Tasks

(a) Construct a concrete numeric example where the overall uplift is +1.2pp even though the segment-level effects are −0.5pp (New) and +2.0pp (Returning), due to a shift in segment mix.

(b) Pre-register a stratified estimator (or MMRM alternative) to estimate a population-average effect that is robust to aggregation bias.

(c) Specify an interaction test for heterogeneity and a decision rule (segment-specific rollout vs. global ship) when heterogeneity is material.

Context

Your goal:

Explain how differences in segment mix across arms can produce aggregation bias (often referred to as Simpson’s paradox in this context).

Pre-register an analysis plan that protects against this bias.

Specify how to test for treatment heterogeneity and how that translates to a rollout decision.

Tasks

(a) Construct a concrete numeric example where the overall uplift is +1.2pp even though the segment-level effects are −0.5pp (New) and +2.0pp (Returning), due to a shift in segment mix.

(b) Pre-register a stratified estimator (or MMRM alternative) to estimate a population-average effect that is robust to aggregation bias.

(c) Specify an interaction test for heterogeneity and a decision rule (segment-specific rollout vs. global ship) when heterogeneity is material.

Detect and address Simpson’s paradox

Quick Overview

Experiment Aggregation Bias and Heterogeneity: Simpson's Paradox, Robust Estimation, and Decisioning

Context

Tasks

Solution

Submit Your Answer to Earn 20XP

Detect and address Simpson’s paradox

Quick Overview

Experiment Aggregation Bias and Heterogeneity: Simpson's Paradox, Robust Estimation, and Decisioning

Context

Tasks

Solution

Submit Your Answer to Earn 20XP