Diagnose a non-significant experiment outcome
Company: Meta
Role: Data Scientist
Category: Statistics & Math
Difficulty: medium
Interview Round: Onsite
You planned α=0.05, 80% power, two‑sample t‑test on the primary mean metric. Baseline=10.0, σ=5.0, planned MDE=+0.6. After 14 days you observe Δ=+0.35 with 95% CI [−0.05, +0.75]. a) Why can this be non‑significant despite large N? b) Should you compute post‑hoc power—if not, what should you report instead and why? c) With a loss function where false positives are twice as costly as false negatives, what’s your decision and next steps (extend sample, reduce variance, or stop)? d) How would you update the design (MDE, variance reduction, duration) for the next iteration?
Quick Answer: This question evaluates understanding of A/B test interpretation, statistical power and confidence intervals, asymmetric loss-aware decision-making, and experimental design adjustments in two-sample hypothesis testing.