A baseline conversion rate is p0 = 0.051 with N0 = 100,000 users. You plan to detect a +5% relative lift (p1 = 1.05 × p0) at α = 0.05 (two-sided) with 80% power. Tasks: 1) Compute a 95% Wald (and an Agresti–Coull or Wilson) confidence interval for p0. Explain why Wilson/AC may be preferable. 2) Approximate the per-variant sample size required for 80% power to detect the target lift using a normal approximation for two proportions. State all formulas and assumptions. 3) If you simultaneously test three metrics (conversion, AOV, retention), apply a Bonferroni or Holm correction and provide the adjusted α for each. Discuss trade-offs vs FDR control (Benjamini–Hochberg). 4) Your data exhibits user-level clustering (repeat visitors). Explain why independence is violated and how to correct standard errors (e.g., cluster-robust SEs or user-level aggregation) and how that affects power.

This question evaluates a candidate's competence in statistical inference for binary outcomes, confidence-interval methods, power and sample-size estimation, multiple-testing corrections, and handling clustered user-level data.

How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

What difficulty level is this interview question?

This is a medium difficulty Statistics & Math question, commonly asked during Onsite rounds at Amazon.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Amazon during technical interviews.

Compute CIs, power, and multiple testing | Amazon Interview Question

A/B Testing Stats: Confidence Intervals, Power, Multiple Testing, and Clustering

Context: You are planning an A/B experiment on a Bernoulli outcome (conversion). The baseline conversion rate is p0 = 0.051 measured on N0 = 100,000 users. You want to detect a +5% relative lift with two-sided α = 0.05 and 80% power.

Given:

Baseline p0 = 0.051, N0 = 100,000
Target lift: p1 = 1.05 × p0
Significance: α = 0.05 (two-sided); Power: 80%

Tasks:

Compute a 95% Wald confidence interval and a Wilson (or Agresti–Coull) interval for p0. Explain why Wilson/AC may be preferable.
Approximate the per-variant sample size required for 80% power to detect the target lift using a normal approximation for two proportions. State all formulas and assumptions.
If you simultaneously test three metrics (conversion, AOV, retention), apply a Bonferroni or Holm correction and provide the adjusted α for each. Discuss trade-offs vs FDR control (Benjamini–Hochberg).
Your data exhibits user-level clustering (repeat visitors). Explain why independence is violated, how to correct standard errors (e.g., cluster-robust SEs or user-level aggregation), and how that affects power.

A/B Testing Stats: Confidence Intervals, Power, Multiple Testing, and Clustering

Given:

Baseline p0 = 0.051, N0 = 100,000

Target lift: p1 = 1.05 × p0

Significance: α = 0.05 (two-sided); Power: 80%

Tasks:

Compute a 95% Wald confidence interval and a Wilson (or Agresti–Coull) interval for p0. Explain why Wilson/AC may be preferable.

Approximate the per-variant sample size required for 80% power to detect the target lift using a normal approximation for two proportions. State all formulas and assumptions.

If you simultaneously test three metrics (conversion, AOV, retention), apply a Bonferroni or Holm correction and provide the adjusted α for each. Discuss trade-offs vs FDR control (Benjamini–Hochberg).

Your data exhibits user-level clustering (repeat visitors). Explain why independence is violated, how to correct standard errors (e.g., cluster-robust SEs or user-level aggregation), and how that affects power.

Compute CIs, power, and multiple testing

Quick Overview

A/B Testing Stats: Confidence Intervals, Power, Multiple Testing, and Clustering

Solution

Comments (0)

Compute CIs, power, and multiple testing

Quick Overview

A/B Testing Stats: Confidence Intervals, Power, Multiple Testing, and Clustering

Solution

Comments (0)