Scenario

A product dashboard shows a sudden drop in a key business KPI immediately after a new release was rolled out to users.

Task

Walk through how you would systematically troubleshoot this unexpected decrease. Describe the specific analyses, validation checks, and decision criteria you would use to determine root cause and next steps.

What to Cover

• Clarify the KPI definition and measurement window.
• Quantify the drop versus an appropriate baseline; assess statistical significance and anomaly detection.
• Segment the impact (e.g., platform, app version, geography, cohorts, traffic source, user tenure).
• Funnel decomposition to localize where the loss occurs.
• Investigate recent changes (release, feature flags, config, data pipelines).
• Use experimentation/rollout data (A/B tests, holdouts, canaries, version-level comparisons).
• Consider external factors (seasonality, outages, policy/market changes).
• Prioritize actions (rollback, hotfix, monitor) based on evidence.

Assume you have standard product analytics logs, an experimentation platform, and a mobile app + web surface.