KPI Diagnosis And Segmentation
Asked of: Data Scientist
Last updated
-
What's being tested
Ability to diagnose metric changes by decomposing KPIs, isolating segments, and using statistical reasoning to distinguish signal from noise. Expect assessment of funnel thinking, instrumentation checks, and causal versus correlational explanations. -
Core knowledge
- KPI decomposition (A = B * C style): express metrics as numerator/denominator products to isolate drivers.
- Funnel analysis: drop-off rates, conversion steps, and stepwise attribution.
- Cohort & time-series comparisons: align cohorts by join date and use seasonality adjustments.
- Segmentation strategy: segment by device, OS, country, traffic source, user cohort, experiment bucket.
- Statistical tests: proportions z-test, t-test, confidence intervals, power, and multiple hypothesis control.
- Instrumentation checks: logging changes, rollout flags, SDK versions, API errors, bot traffic.
- Causal checks: look for concurrent experiments, config changes, external events, and use difference-in-differences.
-
Worked example — “Diagnose a sudden drop in DAU” (framing)
First, verify the signal: compute absolute and relative change, confidence interval, and compare to historical volatility and seasonality. Check instrumentation: recent SDK, event pipeline, sampling, or logging changes. Segment the drop by country, OS, app version, and referrer to localize. Inspect funnels (launch → login → activity) to find which step lost users and correlate with error rates, experiment rollouts, or outages. If localized, estimate impact size and propose short diagnostic queries or quick experiments (rollback, AB test shadowing) to confirm cause. -
A common pitfall
Candidates often jump to product explanations (feature bug) without checking instrumentation or denominator shifts. Another mistake is treating small percentage moves as meaningful without accounting for seasonality, multiple segments, or low sample sizes. This leads to misattribution and unnecessary rollbacks. -
Further reading
- Ron Kohavi et al., "Online Controlled Experiments at Large Scale" (practical guidance on experiments and diagnostics).
Related concepts
- Root Cause Analysis And Segmentation
- Product Metrics, Funnels, And KPI DiagnosisAnalytics & Experimentation
- Product Metric Design And Diagnostic Deep DivesAnalytics & Experimentation
- Product Metrics, Funnels, And SegmentationAnalytics & Experimentation
- Growth Diagnostics, Metric Trees, Estimation, and A/B Testing
- Product Diagnostics And Root Cause Analysis