Compare Bayesian and frequentist decisions

Q: Compare Bayesian and frequentist decisions

This question evaluates understanding of Bayesian inference versus frequentist hypothesis testing, decision theory under asymmetric loss, sequential monitoring/optional stopping, and prior elicitation applied to A/B testing within the Statistics & Math domain.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Statistics & Math question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

A/B Test With Beta–Binomial Posteriors and Decision-Making Under Asymmetric Costs

You ran a two-arm A/B test on a binary KPI with independent Beta(1, 1) priors for each arm. Each arm saw 10,000 users. You observed 515 conversions in arm A and 500 in arm B.

Assume "lift > 0" means that the treatment arm A has a higher true conversion rate than control arm B: Δ = p_A − p_B.

Answer the following:

(a) Compute the posterior distribution for each arm and the posterior probability that lift > 0.

(b) Define a decision rule that minimizes expected loss when the costs of a false positive (shipping a worse variant) and a false negative (failing to ship a better variant) are asymmetric.

(c) Contrast the resulting Bayesian decision with a frequentist two-proportion test, especially under optional stopping. Discuss operational implications (e.g., communicating posterior vs. p‑value, sequential monitoring/alpha spending).

(d) How would you set or elicit priors to avoid over‑optimism in future experiments?

Compare Bayesian and frequentist decisions

Quick Overview

Compare Bayesian and frequentist decisions

A/B Test With Beta–Binomial Posteriors and Decision-Making Under Asymmetric Costs

Write your answer