Design an A/B test; choose Z vs T

Quick Overview

This question evaluates a candidate's competency in experimental design and statistical inference, covering sample-size and power calculations, the choice between Z and T tests, sequential testing/alpha-spending strategies, and causal-inference planning for non-randomized rollouts.

Design an A/B test; choose Z vs T

Company: Uber

Role: Data Scientist

Category: Statistics & Math

Difficulty: hard

Interview Round: Technical Screen

You are planning an A/B test on a signup funnel. Baseline conversion p0 = 0.050, target minimum detectable effect (MDE) = +10% relative (p1 = 0.055). Daily eligible traffic = 50,000 visitors, but only 60% meet targeting criteria and 90% of eligible visitors are successfully bucketed. 1:1 split, two-sided alpha = 0.05, power = 0.80. Assume independence across users. (a) Compute the required sample size per variant and the expected calendar duration (in days) to reach it given the traffic constraints above. State your formula and any continuity or pooled-variance assumptions. (b) Explain whether you would use a Z-test or a T-test for the primary proportion metric and why. Under what conditions do their results materially differ? Include how estimating variance from data, small-sample corrections, and unequal sample sizes/variances affect your choice. (c) If product wants the test to stop “as soon as it looks good,” propose a sequential testing or alpha-spending approach (e.g., group-sequential boundaries) that controls Type I error. Specify stopping rules and how they change the nominal sample size and expected duration. (d) Suppose randomization is not possible for a related rollout. Sketch a causal inference plan: draw a DAG to identify confounders, propose an identification strategy (e.g., difference-in-differences with parallel trends checks or an instrumental variable) and list key assumptions you would need to defend. How would you validate those assumptions in practice?

Quick Answer: This question evaluates a candidate's competency in experimental design and statistical inference, covering sample-size and power calculations, the choice between Z and T tests, sequential testing/alpha-spending strategies, and causal-inference planning for non-randomized rollouts.

|Home/Statistics & Math/Uber
Uber logo
Uber
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenStatistics & Math
9
0

A/B Test on a Signup Funnel: Sample Size, Test Choice, Sequential Design, and Causal Plan

Context

You are planning a two-variant A/B test on a signup funnel with the following parameters:

  • Baseline conversion p0 = 0.050
  • Target minimum detectable effect (MDE) = +10% relative → p1 = 0.055 (Δ = 0.005 absolute)
  • Two-sided alpha = 0.05; power = 0.80
  • Allocation = 1:1
  • Daily site traffic = 50,000 visitors; only 60% meet targeting criteria; 90% of those are successfully bucketed
  • Assume independence across users

Tasks

(a) Compute the required sample size per variant and the expected calendar duration (in days) to reach it given the traffic constraints above. State your formula and any continuity or pooled-variance assumptions.

(b) Explain whether you would use a Z-test or a T-test for the primary proportion metric and why. Under what conditions do their results materially differ? Include how estimating variance from data, small-sample corrections, and unequal sample sizes/variances affect your choice.

(c) If product wants the test to stop “as soon as it looks good,” propose a sequential testing or alpha-spending approach (e.g., group-sequential boundaries) that controls Type I error. Specify stopping rules and how they change the nominal sample size and expected duration.

(d) Suppose randomization is not possible for a related rollout. Sketch a causal inference plan: draw a DAG to identify confounders, propose an identification strategy (e.g., difference-in-differences with parallel trends checks or an instrumental variable) and list key assumptions you would need to defend. How would you validate those assumptions in practice?

Loading comments...