Evaluate Email Subject Line Performance Using Hypotheses
Overview
This question evaluates proficiency with hypothesis testing for proportions, application of the Central Limit Theorem, and power/sample-size calculations in the context of A/B testing email click-through rates.
Uber
Jul 12, 2025, 6:59 PM
Data Scientist
Technical Screen
Statistics & Math
9
0
A/B Test of Email Subject Lines: CTR Hypotheses, CLT Justification, and Sample Size
Context
You are comparing click-through rates (CTRs) between a control email (current subject line) and a test email (new subject line). Each recipient either clicks (1) or not (0), so CTR is a proportion. Assume independent users and equal allocation between variants.
Let:
- p_c = CTR of control
- p_t = CTR of test
- n_c, n_t = sample sizes for control and test (often n_c = n_t = n)
- α = 0.05 significance level, power = 0.80 (β = 0.20)
- Desired minimum detectable effect (MDE) = 2 percentage points = 0.02
Tasks
- State the null and alternative hypotheses for comparing CTRs.
- Explain how the Central Limit Theorem (CLT) justifies a z-test for large samples.
- Derive the sample size needed per group to detect a 2-percentage-point lift with 80% power at α = 0.05. Provide a general formula in terms of the baseline CTR and a numeric example.
Solution
Login required
Show