Diagnose Causes and Test Hypotheses for Metric Drop
Quick Overview
Evaluates incident-style analysis of a sudden purchase-conversion drop in a consumer product. Strong answers list plausible causes, validate with concrete cuts and logs, design an A/B test, and define metrics.
Diagnose Causes and Test Hypotheses for Metric Drop
Company: Amazon
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
The product’s key metric suddenly drops. Stakeholders want a structured diagnosis and experiment plan.
##### Question
List plausible causes for the performance drop, describe analyses you would run to validate each cause, and design an A/B test to confirm the main hypothesis. Which primary and guardrail metrics would you track and why?
##### Hints
Think segmentation, funnel breakouts, external factors, and metric hierarchy (north-star vs. health).
Quick Answer: Evaluates incident-style analysis of a sudden purchase-conversion drop in a consumer product. Strong answers list plausible causes, validate with concrete cuts and logs, design an A/B test, and define metrics.
A large consumer web or mobile product sees its key business metric drop materially and suddenly. Assume sitewide purchase conversion rate fell by 12 percent relative, for example from 10.0 percent to 8.8 percent, starting Tuesday at 10:00 AM and persisting for several days.
Identify plausible causes, validate or falsify each cause with concrete analyses, and design an A/B test for your main hypothesis.
Constraints & Assumptions
Define the metric, denominator, and time window before analysis.
Check data quality and instrumentation before assuming real user behavior changed.
Use segmentation, funnel analysis, release timelines, and external event checks.
Include both primary and guardrail metrics.
Clarifying Questions to Ask Guidance
Did any releases, experiments, traffic changes, marketing campaigns, outages, or pricing changes occur near Tuesday 10:00 AM?
Is the drop visible in server-side sources such as orders or payment logs?
Is the drop concentrated by platform, app version, region, channel, browser, or user cohort?
Was the metric matured and computed with the same eligibility rules as before?
Part 1 - Plausible Causes
List plausible causes for the metric drop.
What This Part Should Cover Guidance
Include instrumentation changes, release bugs, checkout failures, payment issues, latency, inventory, pricing or promotions, traffic mix, marketing changes, fraud rules, external events, seasonality, and experiment interactions.
Separate true product harm from measurement artifacts.
Prioritize causes based on timing and affected segments.
Part 2 - Validation Analyses
For each major cause, describe specific analyses to validate or falsify it.
What This Part Should Cover Guidance
Use cuts by device, geo, channel, app version, browser, cohort, funnel step, and exposure.
Compare client logs to server logs, finance totals, payment processor data, and release metadata.
