Diagnose and fix low conversion rigorously
Overview
This question evaluates a Data Scientist's competency in analytics and experimentation, covering instrumentation validation, funnel and segmentation analysis, causal inference methods (holdouts/DiD), system reliability diagnostics, statistical power and sample-size estimation, and the ability to synthesize findings into executive-level impact statements. It is commonly asked to determine whether a candidate can distinguish measurement or product regressions from marketing-mix shifts and to demonstrate both conceptual understanding of causal inference and practical application of data querying, metric definition, and experiment design in the Analytics & Experimentation domain.
Diagnose a Checkout Conversion Drop After a Promo Banner Launch
Scenario
Week-over-week, checkout conversion fell from 42% to 35% after a new promo banner shipped. Traffic volume is flat, the marketing mix shifted 10% toward paid social, and payment failures rose slightly. You need to isolate the root cause and recommend next steps.
Task
Outline a concise, step-by-step investigation and testing plan that:
- Validates instrumentation and precisely defines the conversion metric.
- Builds a funnel and segments results by device, geo, and new vs. returning users.
- Inspects latency and error/timeout rates across the checkout stack.
- Uses holdouts or a difference-in-differences (DiD) approach to separate promo impact from channel-mix shifts.
- Specifies the minimal queries/metrics to pull to execute the plan.
- Estimates the statistical power and sample size for a rollback or fix-forward A/B test.
- Summarizes the executive-level insight you would present, including expected revenue impact and recommended next actions.
Solution
Login required