Present an A/B test project review
Company: Instacart
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Onsite
## Onsite Project Review: Analyze and present an A/B test
Before the onsite, you completed a take-home project analyzing an **A/B test** (you can assume typical product experimentation data: assignment, exposure, user events, and outcome metrics).
During the onsite, you must present slides and answer deep-dive questions.
### What you should prepare
1. Summarize the experiment goal, design, and key assumptions.
2. Validate experiment integrity and data quality (what checks do you run?).
3. Estimate the treatment effect on pre-specified metrics.
4. Discuss interpretation and limitations (confounding risks, interference, multiple testing, seasonality).
5. Provide a clear ship/no-ship recommendation and next steps.
### Interviewer follow-ups to expect
- What would you do if you see a sample ratio mismatch?
- How do you pick primary vs guardrail metrics?
- How do you handle many metrics or repeated looks at the data?
- What if average impact is neutral but a segment improves a lot?
Overview: This question evaluates a data scientist's competency in experimental design, data integrity checks, statistical estimation of treatment effects, and causal interpretation in A/B testing.
Read the full Instacart Data Scientist interview experience this question came from