Diagnose Causes and Test Hypotheses for Metric Drop

Q: Diagnose Causes and Test Hypotheses for Metric Drop

This question evaluates diagnostic analytics, causal reasoning, instrumentation and data-quality troubleshooting skills along with practical experiment design and statistical power considerations for a Data Scientist role in the Analytics & Experimentation domain.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Scenario

A large consumer web/mobile product sees its key business metric drop materially and suddenly.

Assume: The sitewide purchase conversion rate fell by 12% relative (e.g., 10.0% → 8.8%) starting Tuesday 10:00 AM and persisting for several days.

Task

List plausible causes for the performance drop.
For each cause, describe specific analyses you would run to validate or falsify it (be concrete about cuts, comparisons, and supporting logs/metadata).
Choose your main hypothesis and design an A/B test to confirm it, including:
- Hypothesis and treatment/control design
- Randomization unit and exposure
- Sample size, duration, and analysis plan
Specify primary and guardrail metrics you would track and why.

Hints

Use segmentation (e.g., device, geo, channel, app version), funnel breakouts, external factors, and a metric hierarchy (north-star vs. input vs. health).
Consider data quality and release/flag timelines alongside user behavior.

Diagnose Causes and Test Hypotheses for Metric Drop

Scenario

Task

Hints

Solution

Comments (0)