Calculate a Confidence Interval
Quick Overview
This question evaluates a candidate's competency in statistical inference for A/B testing, specifically the construction and interpretation of confidence intervals for differences in proportions, and falls under the Statistics & Math domain relevant to Data Scientist roles; it tests practical application (deriving point estimates and standard errors) alongside conceptual understanding (assumptions behind normal approximation versus exact, score-based, or bootstrap intervals). Interviewers commonly ask this to assess ability to quantify treatment effects, reason about approximation validity and alternative interval methods, and distinguish between absolute and relative lift when reporting experimental results.
Suppose an online experiment compares treatment and control on conversion rate. Treatment has x1 conversions out of n1 users, and control has x0 conversions out of n0 users.
Show how to compute a 95% confidence interval for the treatment effect by hand.
- Define the point estimate for the absolute lift in conversion rate.
- Derive the standard error under an independent-samples approximation.
- Write the 95% confidence interval formula.
- Explain how the answer changes if the interviewer asks for relative lift instead of absolute lift.
- State when the normal approximation is appropriate and when you would prefer an exact, score-based, or bootstrap interval.