Estimate CTR lift with binomial tests and errors

Q: Estimate CTR lift with binomial tests and errors

This question evaluates statistical inference for proportions, sequential monitoring, and multiple-comparison error control—assessing competence with binomial-based CTR comparison, pooled standard errors, z-tests and confidence intervals, Type I error inflation from peeking, and FWER/FDR considerations.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Statistics & Math question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

A/B Test Inference, Peeking, and Multiple Comparisons

You run a two-arm A/B test of click-through rate (CTR).

Control: n_c = 10,000,000 impressions, CTR_c = 1.20%.
Treatment: n_t = 10,000,000 impressions, CTR_t = 1.23%.

Let p_c and p_t be the CTRs (as proportions).

(a) Compute:

Absolute lift (p_t − p_c) and relative lift (p_t/p_c − 1).
The pooled standard error for the difference in proportions, the z-statistic for H0: p_t = p_c, and the two-sided p-value.
A 95% confidence interval for the relative lift.

(b) Suppose you peeked at significance daily over 14 days using the same test threshold each day (no alpha spending). Quantify the approximate inflation of the overall Type I error, and then propose a corrected sequential monitoring plan using either:

An alpha-spending/group-sequential design, or
A Bayesian sequential alternative with a prior on lift.

(c) You also track 8 guardrail metrics. Explain how you would control familywise error rate (FWER) or false discovery rate (FDR) across these guardrails (and how this interacts with sequential monitoring).

Estimate CTR lift with binomial tests and errors

Quick Overview

A/B Test Inference, Peeking, and Multiple Comparisons

Solution

Comments (0)