##### Scenario A key business metric suddenly spikes or drops; leadership asks for a root-cause analysis and later wants to run an A/B test on a proposed fix. ##### Question How would you systematically investigate the cause of a sudden change in a core metric? 2. Outline the essential steps, assumptions, and success criteria for designing and analyzing an A/B experiment to validate a potential solution. ##### Hints Think root-cause frameworks (segmentation, funnel, time), define control/treatment, randomization, power, sample size, success metric.

This question evaluates a candidate's competency in root-cause analysis and A/B experiment design for diagnosing sudden metric changes, emphasizing skills in data-driven triage, segmentation, decomposition, causal reasoning, and experimentation.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Lyft.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Lyft during technical interviews.

Investigate Sudden Metric Changes and Design A/B Test

Scenario

A core business metric (e.g., conversion, cancellations, or gross bookings) shows a sudden spike or drop. Leadership asks for a rapid root-cause analysis (RCA), and then wants an A/B experiment to validate a proposed fix.

Assume you are the data scientist on a two-sided consumer marketplace with strong diurnal and day-of-week patterns. You have access to product analytics, experimentation logs, feature-flag rollouts, marketing spend, and operational metrics.

Tasks

Root-cause analysis: Describe a systematic approach to investigate the cause of the sudden change. Specify what you would check first (triage), how you would segment the problem, and how you would quantify contributions of different factors.
A/B testing plan: Outline the essential steps, assumptions, and success criteria for designing and analyzing an experiment to validate a potential solution. Include definition of control/treatment, randomization unit, sample size and power, duration, primary/secondary metrics, guardrails, and analysis/decision criteria.

Hints

Root-cause frameworks: segmentation (who/where), funnel (where in the journey), time (when), external factors; rate vs. mix decomposition; change-point detection.
Experiment design: define hypothesis, control/treatment, randomization strategy and unit, handling interference, power/MDE/sample size, duration, instrumentation, guardrails, analysis plan, and rollout criteria.

Scenario

Tasks

Root-cause analysis: Describe a systematic approach to investigate the cause of the sudden change. Specify what you would check first (triage), how you would segment the problem, and how you would quantify contributions of different factors.

A/B testing plan: Outline the essential steps, assumptions, and success criteria for designing and analyzing an experiment to validate a potential solution. Include definition of control/treatment, randomization unit, sample size and power, duration, primary/secondary metrics, guardrails, and analysis/decision criteria.

Hints

Root-cause frameworks: segmentation (who/where), funnel (where in the journey), time (when), external factors; rate vs. mix decomposition; change-point detection.

Experiment design: define hypothesis, control/treatment, randomization strategy and unit, handling interference, power/MDE/sample size, duration, instrumentation, guardrails, analysis plan, and rollout criteria.

Investigate Sudden Metric Changes and Design A/B Test

Quick Overview

Scenario

Tasks

Hints

Solution

Submit Your Answer

Investigate Sudden Metric Changes and Design A/B Test

Quick Overview

Scenario

Tasks

Hints

Solution

Submit Your Answer