Diagnose a non-significant experiment outcome

Quick Overview

This question evaluates understanding of A/B test interpretation, statistical power and confidence intervals, asymmetric loss-aware decision-making, and experimental design adjustments in two-sample hypothesis testing.

Diagnose a non-significant experiment outcome

Company: Meta

Role: Data Scientist

Category: Statistics & Math

Difficulty: medium

Interview Round: Onsite

You planned α=0.05, 80% power, two‑sample t‑test on the primary mean metric. Baseline=10.0, σ=5.0, planned MDE=+0.6. After 14 days you observe Δ=+0.35 with 95% CI [−0.05, +0.75]. a) Why can this be non‑significant despite large N? b) Should you compute post‑hoc power—if not, what should you report instead and why? c) With a loss function where false positives are twice as costly as false negatives, what’s your decision and next steps (extend sample, reduce variance, or stop)? d) How would you update the design (MDE, variance reduction, duration) for the next iteration?

Quick Answer: This question evaluates understanding of A/B test interpretation, statistical power and confidence intervals, asymmetric loss-aware decision-making, and experimental design adjustments in two-sample hypothesis testing.

|Home/Statistics & Math/Meta
Meta logo
Meta
Oct 13, 2025, 9:49 PM
mediumData ScientistOnsiteStatistics & Math
7
0

A/B Test Interpretation, Power, and Decision-Making Under Asymmetric Loss

Context

You ran a two-sample A/B test on a primary mean metric (two-sided t-test). The original design targeted α = 0.05 and 80% power for a minimum detectable effect (MDE) of +0.6 units, assuming a baseline mean of 10.0 and standard deviation σ = 5.0.

After 14 days, you observe an estimated treatment effect Δ = +0.35 with a 95% confidence interval (CI) of [−0.05, +0.75].

Questions

(a) Why can this result be non-significant despite having a large sample size (N)?

(b) Should you compute post‑hoc power? If not, what should you report instead, and why?

(c) Suppose your loss function values false positives (FP) at twice the cost of false negatives (FN). What is your decision now, and what are your next steps (extend sample, reduce variance, or stop)?

(d) How would you update the experimental design (MDE, variance reduction plan, duration/traffic) for the next iteration?

Loading comments...