Interpret p=0.10 rigorously
Company: Other
Role: Data Scientist
Category: Statistics & Math
Difficulty: medium
Interview Round: Technical Screen
An A/B test aims to reduce monthly churn among subscribers. Over 28 days you randomize 150,000 users per arm. Control churn=5.00%; treatment churn=4.85%. A two-sided test yields p=0.10 at α=0.05. Answer: (1) compute the 95% confidence interval for the absolute churn difference (treatment minus control, in percentage points) and interpret it; (2) compute post-hoc power to detect a −0.15 pp effect with these sample sizes; (3) explain why p=0.10 is not “a 10% chance the null is true,” and state what it does mean; (4) under what pre-registered conditions would a one-sided test be appropriate here, what would its p-value be approximately, and why is switching to one-sided post hoc invalid; (5) name two design/analysis changes (e.g., covariate adjustment, stratification, CUPED) that could legitimately reduce variance and potentially move p below 0.05, and when they are valid.
Quick Answer: This question evaluates understanding of hypothesis testing, confidence intervals, power calculations, p-value interpretation, and variance-reduction techniques within A/B testing and experimental design for a Data Scientist role.