Evaluate a new ranking model
Company: DoorDash
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
A food-delivery company currently serves homepage store recommendations with ranking model **V1.1**. A new model **V2.0** adds several new features and may require a different feature configuration for treatment users.
Design an experimentation and rollout plan for this model upgrade.
Your answer should address:
1. How to define the **primary success metric** and important **guardrail metrics** for a homepage recommendation model in a two-sided delivery marketplace.
2. How to choose the **unit of randomization** (for example, user-level, session-level, geo-level, or switchback/time-based) given that recommendations can affect merchant demand, delivery times, and marketplace balance.
3. How the serving infrastructure should support **experiment-specific model versions and feature-set configuration**, so control and treatment groups can fetch different feature lists safely.
4. What events and metadata must be logged so the experiment can be analyzed correctly.
5. How to handle practical issues such as **sample ratio mismatch, delayed conversions, feature missingness, novelty effects, selection bias, and spillover/interference**.
6. How to estimate **power / MDE**, and when methods such as stratification or **CUPED** would help.
7. What criteria you would use for **ramping, rollback, and final launch decisions**.
Quick Answer: This question evaluates a candidate's competency in experimentation design, causal inference and metric definition, logging and analysis, model-serving configuration, and assessment of marketplace impacts for a ranking model upgrade in a two-sided delivery marketplace.