Estimate CTR lift with binomial tests and errors

Quick Overview

This question evaluates statistical inference for proportions, sequential monitoring, and multiple-comparison error control—assessing competence with binomial-based CTR comparison, pooled standard errors, z-tests and confidence intervals, Type I error inflation from peeking, and FWER/FDR considerations.

Estimate CTR lift with binomial tests and errors

Company: Meta

Role: Data Scientist

Category: Statistics & Math

Difficulty: hard

Interview Round: Onsite

In an A B test, control shows 10,000,000 impressions with 1.20% CTR; treatment shows 10,000,000 impressions with 1.23% CTR. (a) Compute the absolute and relative lift, the pooled standard error, z statistic, and two-sided p value. Provide a 95% confidence interval for the relative lift. (b) If you peeked daily over 14 days without alpha spending, quantify approximate Type I error inflation and propose a corrected sequential plan using alpha spending or a Bayesian alternative with a prior on lift. (c) If you also track 8 guardrail metrics, explain how you would control familywise error or false discovery rate.

Quick Answer: This question evaluates statistical inference for proportions, sequential monitoring, and multiple-comparison error control—assessing competence with binomial-based CTR comparison, pooled standard errors, z-tests and confidence intervals, Type I error inflation from peeking, and FWER/FDR considerations.

|Home/Statistics & Math/Meta
Meta logo
Meta
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteStatistics & Math
5
0

A/B Test Inference, Peeking, and Multiple Comparisons

You run a two-arm A/B test of click-through rate (CTR).

  • Control: n_c = 10,000,000 impressions, CTR_c = 1.20%.
  • Treatment: n_t = 10,000,000 impressions, CTR_t = 1.23%.

Let p_c and p_t be the CTRs (as proportions).

(a) Compute:

  1. Absolute lift (p_t − p_c) and relative lift (p_t/p_c − 1).
  2. The pooled standard error for the difference in proportions, the z-statistic for H0: p_t = p_c, and the two-sided p-value.
  3. A 95% confidence interval for the relative lift.

(b) Suppose you peeked at significance daily over 14 days using the same test threshold each day (no alpha spending). Quantify the approximate inflation of the overall Type I error, and then propose a corrected sequential monitoring plan using either:

  • An alpha-spending/group-sequential design, or
  • A Bayesian sequential alternative with a prior on lift.

(c) You also track 8 guardrail metrics. Explain how you would control familywise error rate (FWER) or false discovery rate (FDR) across these guardrails (and how this interacts with sequential monitoring).

Loading comments...