Compare Bayesian and frequentist decisions
Overview
This question evaluates understanding of Bayesian inference versus frequentist hypothesis testing, decision theory under asymmetric loss, sequential monitoring/optional stopping, and prior elicitation applied to A/B testing within the Statistics & Math domain.
A/B Test With Beta–Binomial Posteriors and Decision-Making Under Asymmetric Costs
You ran a two-arm A/B test on a binary KPI with independent Beta(1, 1) priors for each arm. Each arm saw 10,000 users. You observed 515 conversions in arm A and 500 in arm B.
Assume "lift > 0" means that the treatment arm A has a higher true conversion rate than control arm B: Δ = p_A − p_B.
Answer the following:
(a) Compute the posterior distribution for each arm and the posterior probability that lift > 0.
(b) Define a decision rule that minimizes expected loss when the costs of a false positive (shipping a worse variant) and a false negative (failing to ship a better variant) are asymmetric.
(c) Contrast the resulting Bayesian decision with a frequentist two-proportion test, especially under optional stopping. Discuss operational implications (e.g., communicating posterior vs. p‑value, sequential monitoring/alpha spending).
(d) How would you set or elicit priors to avoid over‑optimism in future experiments?
Solution
Login required