Evaluate Email Subject Line Performance Using Hypotheses

Q: Evaluate Email Subject Line Performance Using Hypotheses

This question evaluates proficiency with hypothesis testing for proportions, application of the Central Limit Theorem, and power/sample-size calculations in the context of A/B testing email click-through rates.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Question

A/B Test of Email Subject Lines: CTR Hypotheses, CLT Justification, and Sample Size

Context

You are comparing click-through rates (CTRs) between a control email (current subject line) and a test email (new subject line). Each recipient either clicks (1) or not (0), so CTR is a proportion. Assume independent users and equal allocation between variants.

Let:

p_c = CTR of control
p_t = CTR of test
n_c, n_t = sample sizes for control and test (often n_c = n_t = n)
α = 0.05 significance level, power = 0.80 (β = 0.20)
Desired minimum detectable effect (MDE) = 2 percentage points = 0.02

Tasks

State the null and alternative hypotheses for comparing CTRs.
Explain how the Central Limit Theorem (CLT) justifies a z-test for large samples.
Derive the sample size needed per group to detect a 2-percentage-point lift with 80% power at α = 0.05. Provide a general formula in terms of the baseline CTR and a numeric example.

Evaluate Email Subject Line Performance Using Hypotheses

A/B Test of Email Subject Lines: CTR Hypotheses, CLT Justification, and Sample Size

Context

Tasks

Solution

Comments (0)