Should you roll out if NSM decreases?
Company: Instacart
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
## Scenario
You ran an experiment. The **north star metric (NSM)** is **profit per order**.
### Observed results
- **Average order volume increased** in treatment vs control.
- **Profit per order decreased** (statistically and/or practically meaningfully).
## Task
Should you roll out the change? Explain your decision process.
### Requirements
In your answer, cover:
- Why optimizing the NSM matters vs secondary metrics.
- What additional checks you would run (segment analysis, guardrails, novelty effects, heterogeneous treatment effects).
- When (if ever) you would still consider launching (e.g., if total profit increases, long-term effects, strategic goals).
- A clear final recommendation and next steps.
Overview: This question evaluates proficiency in experimental analysis, metric prioritization, and interpreting trade-offs between a north star metric and secondary signals within A/B testing, testing core data science competencies in the Analytics & Experimentation domain.
Read the full Instacart Data Scientist interview experience this question came from