How to debug an apparent D14 retention drop

Quick Overview

This question evaluates competency in cohort-based retention analysis, understanding metric maturity and right-censoring, performing data quality and definition checks, and localizing real declines across segments, releases, or funnel changes.

How to debug an apparent D14 retention drop

Company: Instacart

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: easy

Interview Round: Technical Screen

## Scenario A dashboard shows **D14 retention** (users retained on day 14 after signup/first activity). In the last week, the chart shows a **sharp decline**. Assume retention is computed as a cohort metric: - Users are assigned to a cohort by **first activity date**. - D14 retention for a cohort is measured 14 days later. ## Task Explain how you would determine whether this decline reflects a real product issue or a **false alarm**. ### Requirements Include: - How metric “maturity” / delayed observation can create misleading recent-week dips. - What plots/tables you would inspect (cohort table, maturity curve, right-censoring). - Data quality checks and definition checks. - If it’s real, how you would localize the cause (segments, releases, funnel changes).

Quick Answer: This question evaluates competency in cohort-based retention analysis, understanding metric maturity and right-censoring, performing data quality and definition checks, and localizing real declines across segments, releases, or funnel changes.

|Home/Analytics & Experimentation/Instacart
Instacart logo
Instacart
Feb 6, 2026, 12:33 PM
easyData ScientistTechnical ScreenAnalytics & Experimentation
7
0

Scenario

A dashboard shows D14 retention (users retained on day 14 after signup/first activity). In the last week, the chart shows a sharp decline.

Assume retention is computed as a cohort metric:

  • Users are assigned to a cohort by first activity date .
  • D14 retention for a cohort is measured 14 days later.

Task

Explain how you would determine whether this decline reflects a real product issue or a false alarm.

Requirements

Include:

  • How metric “maturity” / delayed observation can create misleading recent-week dips.
  • What plots/tables you would inspect (cohort table, maturity curve, right-censoring).
  • Data quality checks and definition checks.
  • If it’s real, how you would localize the cause (segments, releases, funnel changes).
Loading comments...