Decide best email variant using stratified A/B analysis
Company: LinkedIn
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
You ran an email A/B test across two strata (week/location). Week 1 (Los Angeles): Email A sent 100,000 with 10,000 responses; Email B sent 10,000 with 1,500 responses. Week 2 (New York): Email A sent 10,000 with 400 responses; Email B sent 100,000 with 6,000 responses. Decide which variant is better. Requirements: (1) Compute stratum-specific conversion rates and 95% CIs; (2) Test for a common treatment effect across strata using a Mantel–Haenszel estimate (report the common odds ratio and 95% CI) and state the two-sided p-value (alpha = 0.05); (3) Test for effect heterogeneity (e.g., Breslow–Day or an equivalent interaction test) and interpret; (4) Compute the naive pooled difference if you ignore stratification, explain whether Simpson’s paradox occurs here, and why; (5) Make a recommendation (A or B) with justification that reconciles the stratified and naive views. Clearly state any assumptions (e.g., independence, single exposure per user).
Quick Answer: This question evaluates statistical inference and experimentation competencies, focusing on stratified A/B testing, estimation and interpretation of treatment effects (conversion rates, odds ratios, and confidence intervals), and assessment of effect heterogeneity between strata.