Investigate Sudden Metric Changes and Design A/B Test
Overview
This question evaluates a candidate's competency in root-cause analysis and A/B experiment design for diagnosing sudden metric changes, emphasizing skills in data-driven triage, segmentation, decomposition, causal reasoning, and experimentation.
Lyft
Aug 4, 2025, 10:55 AM
Data Scientist
Technical Screen
Analytics & Experimentation
60
0
Scenario
A core business metric (e.g., conversion, cancellations, or gross bookings) shows a sudden spike or drop. Leadership asks for a rapid root-cause analysis (RCA), and then wants an A/B experiment to validate a proposed fix.
Assume you are the data scientist on a two-sided consumer marketplace with strong diurnal and day-of-week patterns. You have access to product analytics, experimentation logs, feature-flag rollouts, marketing spend, and operational metrics.
Tasks
- Root-cause analysis: Describe a systematic approach to investigate the cause of the sudden change. Specify what you would check first (triage), how you would segment the problem, and how you would quantify contributions of different factors.
- A/B testing plan: Outline the essential steps, assumptions, and success criteria for designing and analyzing an experiment to validate a potential solution. Include definition of control/treatment, randomization unit, sample size and power, duration, primary/secondary metrics, guardrails, and analysis/decision criteria.
Hints
- Root-cause frameworks: segmentation (who/where), funnel (where in the journey), time (when), external factors; rate vs. mix decomposition; change-point detection.
- Experiment design: define hypothesis, control/treatment, randomization strategy and unit, handling interference, power/MDE/sample size, duration, instrumentation, guardrails, analysis plan, and rollout criteria.
Solution
Login required
Show