How to debug an apparent D14 retention drop
Company: Instacart
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
## Scenario
A dashboard shows **D14 retention** (users retained on day 14 after signup/first activity). In the last week, the chart shows a **sharp decline**.
Assume retention is computed as a cohort metric:
- Users are assigned to a cohort by **first activity date**.
- D14 retention for a cohort is measured 14 days later.
## Task
Explain how you would determine whether this decline reflects a real product issue or a **false alarm**.
### Requirements
Include:
- How metric “maturity” / delayed observation can create misleading recent-week dips.
- What plots/tables you would inspect (cohort table, maturity curve, right-censoring).
- Data quality checks and definition checks.
- If it’s real, how you would localize the cause (segments, releases, funnel changes).
Overview: This question evaluates competency in cohort-based retention analysis, understanding metric maturity and right-censoring, performing data quality and definition checks, and localizing real declines across segments, releases, or funnel changes.
Read the full Instacart Data Scientist interview experience this question came from