Email Subject Line A/B Test: Hypotheses, CLT, and Sample Size
An email marketing team wants to evaluate whether a new subject line improves click-through rate compared with the current subject line.
Each recipient either clicks or does not click, so the outcome is binary and CTR is a proportion.
Constraints & Assumptions
-
Compare control and test email CTRs.
-
Assume independent recipients and equal allocation unless stated otherwise.
-
State whether the alternative hypothesis is one-sided or two-sided.
-
Derive sample size for detecting a 2 percentage-point lift with 80% power at alpha = 0.05.
-
Provide a formula in terms of the baseline CTR and, if useful, a numeric example.
Clarifying Questions to Ask
-
Is the goal to detect any difference or only an improvement?
-
What is the baseline CTR?
-
Are users randomized once, and can one user receive multiple emails?
-
Are we testing one subject line or multiple variants?
What a Strong Answer Covers
-
Null and alternative hypotheses for comparing two proportions.
-
CLT explanation: sample proportions are approximately normal in large samples, and their difference is approximately normal.
-
Two-proportion z-test setup with pooled standard error under the null.
-
Sample-size formula using baseline
p_c
, target
p_t = p_c + 0.02
, alpha, and power.
-
Clear distinction between one-sided and two-sided tests.
-
Assumptions and checks: enough expected clicks/non-clicks, independent users, stable randomization, no sample ratio mismatch, and no multiple-testing issue unless multiple variants are tested.
Follow-up Questions
-
How would the sample size change if baseline CTR is very low?
-
What if you track opens, clicks, conversions, and unsubscribes?
-
When would you use Fisher's exact test instead of a z-test?
-
How would you adjust for multiple subject-line variants?