Scenario

A core business metric (e.g., conversion, cancellations, or gross bookings) shows a sudden spike or drop. Leadership asks for a rapid root-cause analysis (RCA), and then wants an A/B experiment to validate a proposed fix.

Assume you are the data scientist on a two-sided consumer marketplace with strong diurnal and day-of-week patterns. You have access to product analytics, experimentation logs, feature-flag rollouts, marketing spend, and operational metrics.

Tasks

1. Root-cause analysis: Describe a systematic approach to investigate the cause of the sudden change. Specify what you would check first (triage), how you would segment the problem, and how you would quantify contributions of different factors.
2. A/B testing plan: Outline the essential steps, assumptions, and success criteria for designing and analyzing an experiment to validate a potential solution. Include definition of control/treatment, randomization unit, sample size and power, duration, primary/secondary metrics, guardrails, and analysis/decision criteria.

Hints

• Root-cause frameworks: segmentation (who/where), funnel (where in the journey), time (when), external factors; rate vs. mix decomposition; change-point detection.
• Experiment design: define hypothesis, control/treatment, randomization strategy and unit, handling interference, power/MDE/sample size, duration, instrumentation, guardrails, analysis plan, and rollout criteria.