How to debug an apparent D14 retention drop

Q: How to debug an apparent D14 retention drop

This question evaluates competency in cohort-based retention analysis, understanding metric maturity and right-censoring, performing data quality and definition checks, and localizing real declines across segments, releases, or funnel changes.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Scenario

A dashboard shows D14 retention (users retained on day 14 after signup/first activity). In the last week, the chart shows a sharp decline.

Assume retention is computed as a cohort metric:

Users are assigned to a cohort by first activity date .
D14 retention for a cohort is measured 14 days later.

Task

Explain how you would determine whether this decline reflects a real product issue or a false alarm.

Requirements

Include:

How metric “maturity” / delayed observation can create misleading recent-week dips.
What plots/tables you would inspect (cohort table, maturity curve, right-censoring).
Data quality checks and definition checks.
If it’s real, how you would localize the cause (segments, releases, funnel changes).

How to debug an apparent D14 retention drop

Quick Overview

Scenario

Task

Requirements

Solution

Comments (0)