Compute power and interpret uplift metrics

Quick Overview

This question evaluates a Data Scientist's practical and conceptual competencies in A/B test design and analysis within the Statistics & Math domain, covering power/sample-size calculations, binary outcome inference, CUPED variance reduction, multiple-testing adjustments, and handling clustered/geographical data.

Compute power and interpret uplift metrics

Company: Stripe

Role: Data Scientist

Category: Statistics & Math

Difficulty: medium

Interview Round: Technical Screen

Two‑arm experiment on conversion. Baseline conversion p0 = 0.120. You want to detect an absolute uplift of Δ = +0.005 (two‑sided α = 0.05, power = 0.80). (1) Approximate the required per‑variant sample size using the normal approximation for a difference in proportions; show your formula and calculation. (2) You ran for a week and observed: Control nC = 150,000, xC = 18,000; Treatment nT = 150,000, xT = 18,900. Compute the point estimate Δ̂, a 95% confidence interval, and a p‑value; interpret business significance vs statistical significance. (3) With CUPED using a pre‑period covariate yielding R² = 0.30 on the outcome, estimate the new effective sample size or variance and the revised MDE; show your math. (4) You track 4 guardrails with unadjusted p‑values {0.03, 0.01, 0.20, 0.04}. Apply Holm–Bonferroni at familywise α = 0.05 and state which guardrails remain significant; show ordering and adjusted thresholds. (5) Explain how you would check and correct for overdispersion or miscalibration in conversion estimates when there is user clustering by geo.

Quick Answer: This question evaluates a Data Scientist's practical and conceptual competencies in A/B test design and analysis within the Statistics & Math domain, covering power/sample-size calculations, binary outcome inference, CUPED variance reduction, multiple-testing adjustments, and handling clustered/geographical data.

|Home/Statistics & Math/Stripe
Stripe logo
Stripe
Oct 13, 2025, 9:49 PM
mediumData ScientistTechnical ScreenStatistics & Math
2
0

A/B Test on Conversion: Powering, Inference, CUPED, Multiple Testing, and Clustering

You are running a two-arm A/B experiment on a binary conversion outcome.

Assume a baseline conversion p0 = 0.120 and a target absolute uplift of Δ = +0.005. Use a two-sided test with α = 0.05 and power = 0.80 unless stated otherwise.

  1. Sample size planning
  • Using the normal approximation for a difference in proportions, derive and compute the required per-variant sample size. Show the formula and numeric calculation.
  1. Post-experiment inference
  • After one week you observe: Control nC = 150,000, xC = 18,000; Treatment nT = 150,000, xT = 18,900.
  • Compute the point estimate Δ^\hat{\Delta} , a 95% confidence interval for the difference, and a two-sided p-value. Interpret statistical significance vs business significance.
  1. CUPED variance reduction
  • Using a pre-period covariate with CUPED that yields R² = 0.30 on the outcome, estimate the new effective sample size (or variance) and the revised MDE for the same design. Show your math and state assumptions.
  1. Multiple testing with guardrails
  • You track 4 guardrail metrics with unadjusted p-values {0.03, 0.01, 0.20, 0.04}.
  • Apply the Holm–Bonferroni method at familywise α = 0.05. Show the ordering, adjusted thresholds for each step, and which guardrails remain significant.
  1. Clustering by geography
  • Explain how you would check for and correct overdispersion or miscalibration in conversion estimates when there is user clustering by geo.
Loading comments...