Interpret p=0.10 rigorously

Q: Interpret p=0.10 rigorously

This is a Statistics & Math interview question from Other for Data Scientist roles. View the full question and solution on PracHub.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Question

A/B Test on Monthly Churn: Inference, Power, and Testing Choices

You ran a 28-day A/B test to reduce monthly churn among subscribers, randomizing 150,000 users per arm. The observed churn rates were:

Control: 5.00%
Treatment: 4.85%

A pre-specified two-sided test reported p = 0.10 at α = 0.05.

Tasks:

Compute the 95% confidence interval for the absolute churn difference (treatment − control, in percentage points) and interpret it.
Compute the post-hoc power to detect a −0.15 pp effect with these sample sizes.
Explain why p = 0.10 is not “a 10% chance the null is true,” and state what it does mean.
State when a one-sided test would be appropriate if pre-registered, approximate its p-value here, and explain why switching to one-sided post hoc is invalid.
Name two design/analysis changes (e.g., covariate adjustment, stratification, CUPED) that could legitimately reduce variance and potentially move p below 0.05, and when they are valid.

Interpret p=0.10 rigorously

A/B Test on Monthly Churn: Inference, Power, and Testing Choices

Solution (Locked)

Comments (0)