Evaluate Email Subject Line Performance Using Hypotheses
Company: Uber
Role: Data Scientist
Category: Statistics & Math
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
Email marketing team wants to evaluate the performance of a new subject line.
##### Question
Define the null and alternative hypotheses for comparing click-through rates between control and test emails. Explain how the Central Limit Theorem justifies using a z-test in large samples. Derive the sample size needed to detect a 2-percentage-point lift with 80% power at α = 0.05.
##### Hints
Think proportions, pooled variance, power formula.
Quick Answer: Uber statistics prompt on email subject-line A/B testing, covering CTR hypotheses, Central Limit Theorem justification, two-proportion z-tests, sample size for 2 percentage-point lift, power, and assumptions.
Evaluate Email Subject Line Performance Using Hypotheses
Uber
Jul 12, 2025, 6:59 PM
mediumData ScientistTechnical ScreenStatistics & Math
17
0
Email Subject Line A/B Test: Hypotheses, CLT, and Sample Size
An email marketing team wants to evaluate whether a new subject line improves click-through rate compared with the current subject line.
Each recipient either clicks or does not click, so the outcome is binary and CTR is a proportion.
Constraints & Assumptions
Compare control and test email CTRs.
Assume independent recipients and equal allocation unless stated otherwise.
State whether the alternative hypothesis is one-sided or two-sided.
Derive sample size for detecting a 2 percentage-point lift with 80% power at alpha = 0.05.
Provide a formula in terms of the baseline CTR and, if useful, a numeric example.
Clarifying Questions to Ask Guidance
Is the goal to detect any difference or only an improvement?
What is the baseline CTR?
Are users randomized once, and can one user receive multiple emails?
Are we testing one subject line or multiple variants?
What a Strong Answer Covers Guidance
Null and alternative hypotheses for comparing two proportions.
CLT explanation: sample proportions are approximately normal in large samples, and their difference is approximately normal.
Two-proportion z-test setup with pooled standard error under the null.
Sample-size formula using baseline
p_c
, target
p_t = p_c + 0.02
, alpha, and power.
Clear distinction between one-sided and two-sided tests.
Assumptions and checks: enough expected clicks/non-clicks, independent users, stable randomization, no sample ratio mismatch, and no multiple-testing issue unless multiple variants are tested.
Follow-up Questions Guidance
How would the sample size change if baseline CTR is very low?
What if you track opens, clicks, conversions, and unsubscribes?
When would you use Fisher's exact test instead of a z-test?
How would you adjust for multiple subject-line variants?