Evaluate an email test with confounding

# Evaluate an Email Test With Confounding A marketing team tested Email A and Email B across two cities, San Francisco and New York, over two weeks. The raw results appear contradictory: - Within each city or week, Email B seems to outperform Email A. - When all observations are combined, Email A appears better overall. How would you determine which email is actually better? ### Constraints & Assumptions - Treat this as a possible Simpson's paradox or confounding problem. - Define the business metric before choosing the winner. - Consider city, week, send time, time zone, user segment, and allocation imbalance. - Discuss whether and how you can compute confidence intervals. - If the test design is flawed, explain how you would rerun it. ### Clarifying Questions to Ask - What is the primary metric: open rate, click-through rate, conversion, revenue, or retention? - Was assignment randomized within each city and week? - Were sample sizes balanced across treatment groups and strata? - Were there other campaign changes, targeting rules, or delivery-time differences? ### What a Strong Answer Covers - Recognize Simpson's paradox: aggregate results can reverse subgroup results when treatment allocation differs across strata with different baselines. - Build a city by week by email table with exposure, conversions, rates, standard errors, and confidence intervals. - Check randomization, balance, sample ratio mismatch, seasonality, targeting, time-zone effects, and user mix. - Estimate a treatment effect with stratification, weighted averages, or regression controlling for city and week. - Use confidence intervals or hypothesis tests if assumptions are valid and sample sizes are sufficient. - Prefer a rerun or redesign if assignment was not randomized or key confounders are uncontrolled. - Include guardrails such as unsubscribe rate, spam complaints, and downstream conversion quality. ### Follow-up Questions - What would you report if B wins in every stratum but A wins overall? - How would you choose weights when combining city-level effects? - Can you compute a confidence interval from only aggregated data? - How would you design the next experiment to avoid this issue?