Design experiments and diagnose metric changes
Company: Meta
Role: Product Analyst
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
You are a Product/Data Scientist at a food-delivery marketplace (customers, dashers/couriers, merchants). Answer the following product analytics & experimentation prompts.
## Scenario A — “Top Dasher” program change
The company is considering a change to the **Top Dasher** program (a set of incentives/benefits intended to improve dasher supply and delivery quality).
1. List **pros/cons** of launching or expanding such a program.
2. Define a **metric framework** for evaluation:
- Primary success metric(s)
- Diagnostic metrics
- Guardrails (e.g., cost, quality, fairness)
3. What is the **randomization unit** for an experiment (dasher vs market vs time vs geo), and why? Discuss trade-offs like interference/spillovers.
## Scenario B — Your test metric is worse than control
In an A/B test, the primary metric is **lower in treatment than control**.
1. What are the first checks you do before concluding the change is harmful?
2. How do you decide whether to stop, continue, or iterate the experiment?
3. What follow-up analyses help you understand *why* it got worse?
## Scenario C — Order cancellation rate is high
The **order cancellation rate** has increased substantially.
1. How do you diagnose the problem end-to-end (data + product)?
2. Which **orgs/systems** (merchant ops, courier ops, pricing, dispatch, support, payments, app reliability, etc.) are likely impacted?
3. Propose hypotheses for root causes and describe how you would **test** them (experiments or quasi-experiments).
## Scenario D — Merchant promotions: self-serve vs auto setup
The company is deciding between:
- **Self-serve promotions:** merchants configure their own discounts/promotions, or
- **Auto setup:** the platform automatically recommends/sets promotions.
1. Compare **pros/cons** for merchants, customers, and the platform.
2. Propose an **experiment plan** (success metrics, guardrails, duration).
3. Choose a **randomization unit** and discuss trade-offs (contamination, fairness, heterogeneous effects).
Quick Answer: This question evaluates product analytics and experimentation skills, including experiment design, metric framework definition, causal reasoning, diagnostic analysis, and marketplace operations trade-offs within the Analytics & Experimentation domain.