Decide best email variant using stratified A/B analysis

Quick Overview

This question evaluates statistical inference and experimentation competencies, focusing on stratified A/B testing, estimation and interpretation of treatment effects (conversion rates, odds ratios, and confidence intervals), and assessment of effect heterogeneity between strata.

Decide best email variant using stratified A/B analysis

Company: LinkedIn

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

You ran an email A/B test across two strata (week/location). Week 1 (Los Angeles): Email A sent 100,000 with 10,000 responses; Email B sent 10,000 with 1,500 responses. Week 2 (New York): Email A sent 10,000 with 400 responses; Email B sent 100,000 with 6,000 responses. Decide which variant is better. Requirements: (1) Compute stratum-specific conversion rates and 95% CIs; (2) Test for a common treatment effect across strata using a Mantel–Haenszel estimate (report the common odds ratio and 95% CI) and state the two-sided p-value (alpha = 0.05); (3) Test for effect heterogeneity (e.g., Breslow–Day or an equivalent interaction test) and interpret; (4) Compute the naive pooled difference if you ignore stratification, explain whether Simpson’s paradox occurs here, and why; (5) Make a recommendation (A or B) with justification that reconciles the stratified and naive views. Clearly state any assumptions (e.g., independence, single exposure per user).

Quick Answer: This question evaluates statistical inference and experimentation competencies, focusing on stratified A/B testing, estimation and interpretation of treatment effects (conversion rates, odds ratios, and confidence intervals), and assessment of effect heterogeneity between strata.

|Home/Analytics & Experimentation/LinkedIn
LinkedIn logo
LinkedIn
Oct 13, 2025, 9:49 PM
mediumData ScientistTechnical ScreenAnalytics & Experimentation
7
0

Stratified A/B Test Across Two Strata (Week/Location)

You ran an email A/B test across two strata defined by week/location. Each user receives at most one email and either responds (success) or not (failure).

Strata and counts:

  • Stratum 1 — Week 1 (Los Angeles):
    • A: 100,000 sent, 10,000 responses
    • B: 10,000 sent, 1,500 responses
  • Stratum 2 — Week 2 (New York):
    • A: 10,000 sent, 400 responses
    • B: 100,000 sent, 6,000 responses

Tasks:

  1. Compute stratum-specific conversion rates (A and B) and 95% confidence intervals.
  2. Using a Mantel–Haenszel (MH) approach, estimate the common treatment effect across strata: report the common odds ratio (OR) with 95% CI and a two-sided p-value (alpha = 0.05).
  3. Test for effect heterogeneity across strata (e.g., Breslow–Day or an equivalent interaction test) and interpret.
  4. Compute the naive pooled difference in conversion (ignoring stratification) and state whether Simpson’s paradox occurs, and why.
  5. Recommend A or B, reconciling the stratified and naive results.

Assume standard A/B test conditions unless stated otherwise (e.g., independent Bernoulli outcomes, single exposure per user, no interference, and randomization within each stratum).

Loading comments...