Design and Interpret an A/B Test
Company: Instacart
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
You are evaluating a product experiment with a control group and two treatment variants, v1 and v2. The primary metric is a user-level conversion rate measured over a fixed exposure window. Assume users are randomly assigned, the planned experiment length is 14 days, and the analysis uses a two-sided test at significance level 0.05 unless otherwise stated.
Answer the following:
1. How would you determine the required experiment duration? What inputs are needed?
2. If the experiment was planned to run for 14 days, but after 3 days the p-value for the primary metric is below 0.05, can you stop the experiment and launch the winning variant? Why or why not?
3. Suppose you are given three separate confidence intervals: one for the control group's conversion rate, one for v1, and one for v2. Can you make a launch decision from these intervals alone? What additional analysis or information would you need?
Quick Answer: This question evaluates expertise in experimental design and statistical inference for A/B testing—specifically sample size and duration calculation, hypothesis testing and stopping rules, interpretation of p-values and confidence intervals, and handling multiple comparisons—within the Analytics & Experimentation domain for a Data Scientist role.