How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Instacart.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Instacart during technical interviews.

Compute duration and stopping rules correctly | Instacart Interview Question

Quick Overview

This question evaluates a data scientist's competency in experimental design and applied statistics, specifically power and sample‑size calculations, accounting for intra‑household clustering and design effects, sequential monitoring for peeking control, and variance‑reduction or stratified randomization to handle day‑of‑week seasonality.

Experiment Runtime Planning: Same‑Day Delivery Attach Rate

You are planning an A/B test on the same‑day delivery attach rate (the proportion of eligible checkouts that attach same‑day delivery). Historical attach rate is 22%. You want to detect a minimum detectable effect (MDE) of a 5% relative lift. Use a two‑sided α=0.05 and power=0.8.

Context and constraints:

The unit of analysis is an eligible checkout.
Daily average eligible checkouts = 120k (sitewide traffic is larger; only ~20% are eligible). Eligibility and traffic vary by weekday with strong Sunday spikes.
There is clustering at the household level; assume a 10% design effect inflation due to intra‑household correlation.

Tasks

Sample size and duration
- Compute the required sample size per arm.
- Translate the requirement into calendar days given 50/50 allocation, day‑of‑week (DoW) seasonality, and the 10% design effect.
Peeking‑safe monitoring plan
- Specify a sequential monitoring plan (e.g., O'Brien–Fleming alpha‑spending or group‑sequential design) to allow interim looks without inflating Type I error.
- Explain how this plan changes the expected duration vs. a fixed‑horizon test.
Handling Sunday volatility (Miami)
- Propose either DoW/geo‑stratified randomization or CUPED‑style variance reduction (or both) to handle volatile Sundays.
- Quantify the expected variance reduction and the resulting impact on required sample size and duration.

Quick Overview

Experiment Runtime Planning: Same‑Day Delivery Attach Rate

Context and constraints:

The unit of analysis is an eligible checkout.

Daily average eligible checkouts = 120k (sitewide traffic is larger; only ~20% are eligible). Eligibility and traffic vary by weekday with strong Sunday spikes.

There is clustering at the household level; assume a 10% design effect inflation due to intra‑household correlation.

Tasks

Sample size and duration

Compute the required sample size per arm.
Translate the requirement into calendar days given 50/50 allocation, day‑of‑week (DoW) seasonality, and the 10% design effect.

Peeking‑safe monitoring plan

Specify a sequential monitoring plan (e.g., O'Brien–Fleming alpha‑spending or group‑sequential design) to allow interim looks without inflating Type I error.
Explain how this plan changes the expected duration vs. a fixed‑horizon test.

Handling Sunday volatility (Miami)

Propose either DoW/geo‑stratified randomization or CUPED‑style variance reduction (or both) to handle volatile Sundays.
Quantify the expected variance reduction and the resulting impact on required sample size and duration.

Compute duration and stopping rules correctly

Quick Overview

Experiment Runtime Planning: Same‑Day Delivery Attach Rate

Solution

Comments (0)

Compute duration and stopping rules correctly

Quick Overview

Experiment Runtime Planning: Same‑Day Delivery Attach Rate

Solution

Comments (0)