A/B Test Decision: Frequentist, Conditional Power, Bayesian, and Risk-Mitigation
Context
You ran a 50/50 A/B test on 30‑day subscriber retention. The pre‑registered analysis plan specified:
-
Two‑sided α = 0.05
-
Power = 80%
-
MDE = +2% relative lift in 30‑day retention
After 4 weeks:
-
Observed effect: +1.4% relative lift
-
Two‑sided p‑value: 0.10 (guardrails show no harm)
-
Business costs: false positives cost 3× false negatives
Assume equal allocation, large‑sample normal approximations are valid, and metric maturity is properly handled for the observed p‑value.
Tasks
-
Under the pre‑registered frequentist plan, give a binary launch/no‑launch decision and justify.
-
Compute or outline the conditional power and the additional sample size/duration needed to achieve 80% power for a 1.4% relative lift; recommend extend vs stop.
-
Reframe with a simple Bayesian analysis (weakly informative prior) and compare expected loss for launch vs no‑launch under the 3:1 cost ratio.
-
If shipping despite p = 0.10, propose a risk‑mitigated rollout (phased ramp, sequential testing/CUPED, kill switches) and explicit rollback criteria.