Diagnose Causes and Test Hypotheses for Metric Drop
Overview
This question evaluates diagnostic analytics, causal reasoning, instrumentation and data-quality troubleshooting skills along with practical experiment design and statistical power considerations for a Data Scientist role in the Analytics & Experimentation domain.
Amazon
Jul 12, 2025, 6:59 PM
Data Scientist
Technical Screen
Analytics & Experimentation
28
0
Scenario
A large consumer web/mobile product sees its key business metric drop materially and suddenly.
Assume: The sitewide purchase conversion rate fell by 12% relative (e.g., 10.0% → 8.8%) starting Tuesday 10:00 AM and persisting for several days.
Task
- List plausible causes for the performance drop.
- For each cause, describe specific analyses you would run to validate or falsify it (be concrete about cuts, comparisons, and supporting logs/metadata).
-
Choose your main hypothesis and design an A/B test to confirm it, including:
- Hypothesis and treatment/control design
- Randomization unit and exposure
- Sample size, duration, and analysis plan
- Specify primary and guardrail metrics you would track and why.
Hints
- Use segmentation (e.g., device, geo, channel, app version), funnel breakouts, external factors, and a metric hierarchy (north-star vs. input vs. health).
- Consider data quality and release/flag timelines alongside user behavior.
Solution
Login required
Show