Design experiments and diagnose metric changes

You are a Product/Data Scientist at a food-delivery marketplace (customers, dashers/couriers, merchants). Answer the following product analytics & experimentation prompts. ## Scenario A — “Top Dasher” program change The company is considering a change to the **Top Dasher** program (a set of incentives/benefits intended to improve dasher supply and delivery quality). 1. List **pros/cons** of launching or expanding such a program. 2. Define a **metric framework** for evaluation: - Primary success metric(s) - Diagnostic metrics - Guardrails (e.g., cost, quality, fairness) 3. What is the **randomization unit** for an experiment (dasher vs market vs time vs geo), and why? Discuss trade-offs like interference/spillovers. ## Scenario B — Your test metric is worse than control In an A/B test, the primary metric is **lower in treatment than control**. 1. What are the first checks you do before concluding the change is harmful? 2. How do you decide whether to stop, continue, or iterate the experiment? 3. What follow-up analyses help you understand *why* it got worse? ## Scenario C — Order cancellation rate is high The **order cancellation rate** has increased substantially. 1. How do you diagnose the problem end-to-end (data + product)? 2. Which **orgs/systems** (merchant ops, courier ops, pricing, dispatch, support, payments, app reliability, etc.) are likely impacted? 3. Propose hypotheses for root causes and describe how you would **test** them (experiments or quasi-experiments). ## Scenario D — Merchant promotions: self-serve vs auto setup The company is deciding between: - **Self-serve promotions:** merchants configure their own discounts/promotions, or - **Auto setup:** the platform automatically recommends/sets promotions. 1. Compare **pros/cons** for merchants, customers, and the platform. 2. Propose an **experiment plan** (success metrics, guardrails, duration). 3. Choose a **randomization unit** and discuss trade-offs (contamination, fairness, heterogeneous effects).