Compute sample size and analyze A/B results

Quick Overview

This question evaluates competency in experimental design and statistical inference for A/B testing, covering two-proportion sample size calculation, sequential correction methods (e.g.

Compute sample size and analyze A/B results

Company: Microsoft

Role: Data Scientist

Category: Statistics & Math

Difficulty: medium

Interview Round: Onsite

Baseline conversion is 5%. You want 90% power at α=0.05 (two-sided) to detect a 6% relative lift. Compute per-variant sample size for a Bernoulli outcome using a normal approximation, then adjust for 10% bot traffic and a 7-day window assuming independence by day. After running: A has 50,000 users and 2,650 conversions; B has 49,500 users and 2,820 conversions, with one interim look at day 3. Calculate the p-value and 95% CI for the difference in proportions, correct for the interim look (e.g., O’Brien–Fleming or Bonferroni), check for sample-ratio mismatch, and conclude whether to ship while discussing risks of Type S/M errors and baseline mis-specification.

Quick Answer: This question evaluates competency in experimental design and statistical inference for A/B testing, covering two-proportion sample size calculation, sequential correction methods (e.g.

|Home/Statistics & Math/Microsoft
Microsoft logo
Microsoft
Oct 13, 2025, 9:49 PM
mediumData ScientistOnsiteStatistics & Math
9
0

A/B Test: Sample Size, Sequential Correction, and Post-Experiment Analysis

Context

You are planning a two-arm A/B test with a binary (Bernoulli) conversion outcome and equal allocation. The baseline conversion rate is 5%. You want 90% power at a two-sided α = 0.05 to detect a 6% relative lift. Use a normal approximation for sample size.

Assumptions (made explicit for clarity):

  • "6% relative lift" means p1 = 1.06 × p0.
  • Allocation is 50/50; outcome is per-user Bernoulli within the measurement window.
  • 10% of traffic is bots and will be excluded (or is non-informative), so gross traffic must be inflated.
  • The test runs for 7 days; assume independent daily increments to translate total sample to an approximate per-day requirement and to define the information fraction for a single interim look at day 3 (t = 3/7).

Tasks

  1. Compute per-variant sample size using a two-proportion z-test normal approximation.
  2. Adjust the required gross traffic for 10% bot share; translate to a 7-day window assuming independence by day (i.e., per-day need).
  3. After running, you observe:
    • Variant A: nA = 50,000 users; xA = 2,650 conversions.
    • Variant B: nB = 49,500 users; xB = 2,820 conversions. There was one interim look at day 3.
    • Compute the p-value and a 95% CI for the difference in proportions.
    • Correct for the interim look using an O’Brien–Fleming spending function (t1 = 3/7) or Bonferroni.
    • Check for a sample-ratio mismatch (SRM).
    • Conclude whether to ship, discussing Type S/M risks and baseline mis-specification.
Loading comments...