Act when A/B result is not significant

An A/B test for the 60s video change yields a non-significant lift on the primary metric at α = 0.05. Baseline completion rate is 22%. Product’s minimum meaningful effect is +5% relative (to 23.1%). At 80% power, two-sided test, equal allocation: (1) compute the required per-arm sample size and explain your formula/assumptions; (2) propose variance reduction (e.g., CUPED with pre-period completion, covariate adjustment) and estimate the power gain if R² = 0.25; (3) decide whether to continue collecting data, stop, or pivot to a Bayesian decision with a utility-based threshold; (4) discuss multiplicity control if multiple related outcomes are monitored; and (5) outline a sequential design (e.g., O’Brien–Fleming) that would have preserved Type I error while allowing early looks.