Compute duration and stopping rules correctly
Overview
This question evaluates a data scientist's competency in experimental design and applied statistics, specifically power and sample‑size calculations, accounting for intra‑household clustering and design effects, sequential monitoring for peeking control, and variance‑reduction or stratified randomization to handle day‑of‑week seasonality.
Instacart
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Experiment Runtime Planning: Same‑Day Delivery Attach Rate
You are planning an A/B test on the same‑day delivery attach rate (the proportion of eligible checkouts that attach same‑day delivery). Historical attach rate is 22%. You want to detect a minimum detectable effect (MDE) of a 5% relative lift. Use a two‑sided α=0.05 and power=0.8.
Context and constraints:
- The unit of analysis is an eligible checkout.
- Daily average eligible checkouts = 120k (sitewide traffic is larger; only ~20% are eligible). Eligibility and traffic vary by weekday with strong Sunday spikes.
- There is clustering at the household level; assume a 10% design effect inflation due to intra‑household correlation.
Tasks
-
Sample size and duration
- Compute the required sample size per arm.
- Translate the requirement into calendar days given 50/50 allocation, day‑of‑week (DoW) seasonality, and the 10% design effect.
-
Peeking‑safe monitoring plan
- Specify a sequential monitoring plan (e.g., O'Brien–Fleming alpha‑spending or group‑sequential design) to allow interim looks without inflating Type I error.
- Explain how this plan changes the expected duration vs. a fixed‑horizon test.
-
Handling Sunday volatility (Miami)
- Propose either DoW/geo‑stratified randomization or CUPED‑style variance reduction (or both) to handle volatile Sundays.
- Quantify the expected variance reduction and the resulting impact on required sample size and duration.
Solution
Login required
Show