Diagnosing and Testing a Sudden Metric Drop
A large consumer web or mobile product sees its key business metric drop materially and suddenly. Assume sitewide purchase conversion rate fell by 12 percent relative, for example from 10.0 percent to 8.8 percent, starting Tuesday at 10:00 AM and persisting for several days.
Identify plausible causes, validate or falsify each cause with concrete analyses, and design an A/B test for your main hypothesis.
Constraints & Assumptions
-
Define the metric, denominator, and time window before analysis.
-
Check data quality and instrumentation before assuming real user behavior changed.
-
Use segmentation, funnel analysis, release timelines, and external event checks.
-
Include both primary and guardrail metrics.
Clarifying Questions to Ask
-
Did any releases, experiments, traffic changes, marketing campaigns, outages, or pricing changes occur near Tuesday 10:00 AM?
-
Is the drop visible in server-side sources such as orders or payment logs?
-
Is the drop concentrated by platform, app version, region, channel, browser, or user cohort?
-
Was the metric matured and computed with the same eligibility rules as before?
Part 1 - Plausible Causes
List plausible causes for the metric drop.
What This Part Should Cover
-
Include instrumentation changes, release bugs, checkout failures, payment issues, latency, inventory, pricing or promotions, traffic mix, marketing changes, fraud rules, external events, seasonality, and experiment interactions.
-
Separate true product harm from measurement artifacts.
-
Prioritize causes based on timing and affected segments.
Part 2 - Validation Analyses
For each major cause, describe specific analyses to validate or falsify it.
What This Part Should Cover
-
Use cuts by device, geo, channel, app version, browser, cohort, funnel step, and exposure.
-
Compare client logs to server logs, finance totals, payment processor data, and release metadata.
-
Check funnel drop-off, error logs, latency, A/B assignments, SRM, traffic composition, and external calendars.
-
Quantify effect size by segment and time.
Part 3 - A/B Test for Main Hypothesis
Choose your main hypothesis and design an A/B test to confirm it.
What This Part Should Cover
-
Define treatment, control, randomization unit, eligibility, exposure, sample size, duration, and analysis plan.
-
Use rollback, fix-forward, holdout, or feature-flag comparison as appropriate.
-
Include power, MDE, statistical tests, and pre-defined decision thresholds.
-
Protect users if the suspected bug has severe business impact.
Part 4 - Metrics
Specify primary and guardrail metrics.
What This Part Should Cover
-
Primary metrics may include purchase conversion, order count, revenue per visitor, and checkout completion.
-
Guardrails may include latency, errors, payment success, refunds, support contacts, retention, and user experience.
-
Track both leading indicators and lagging business outcomes.
Follow-up Questions
-
What would you do in the first 30 minutes after the drop is detected?
-
How would you decide whether to roll back before statistical confirmation?
-
What if the metric recovers before you finish the analysis?
