You are the data scientist for a food-delivery marketplace. Customers are reporting that some orders arrive cold. The product team proposes an intervention intended to keep food warm during delivery and asks whether it should be launched broadly. Explain how you would diagnose the problem, define success, evaluate the intervention, and decide whether to launch it. Be explicit about the data you would inspect and the comparisons you would make. If a randomized A/B test is not possible, propose a credible alternative. ### Constraints & Assumptions - Treat this as a marketplace with customers, merchants, and couriers. A decision that improves one side by seriously harming another is not an acceptable success. - The phrase "cold food" is not yet an operational metric. You may propose a definition, but state its limitations. - The intervention is unspecified on purpose. Explain which parts of your design depend on whether it operates at the order, courier, merchant, or market level. - You have order events, promised and actual delivery timestamps, support contacts, refunds, ratings, merchant and courier identifiers, geography, item attributes, and experiment-exposure logs when experimentation is available. - Do not assume that a complaint or refund is a perfect label for food temperature. ### Clarifying Questions to Ask - What intervention is being considered, and at what unit can it be assigned? - Which users, merchants, cuisines, order distances, or markets appear most affected? - Do we have a direct temperature signal, or only proxies such as complaint reasons, refunds, ratings, and delivery time? - Is the goal to reduce cold-food incidents, improve retention, reduce support cost, or balance several objectives? - What decision deadline, rollout risk, minimum meaningful effect, and observation window apply? ### Part 1: Diagnose and Measure the Problem Describe how you would turn the vague report of "cold food" into a measurable outcome. Lay out the analyses you would run to estimate prevalence, find where the problem occurs, and distinguish plausible operational causes from reporting artifacts. **Hint:** Start with the measurement process and the delivery funnel before choosing a statistical model. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 2: Choose Primary, Secondary, and Guardrail Metrics Propose a metric hierarchy for evaluating the intervention. Define one primary metric, useful secondary metrics, and guardrails for customers, merchants, couriers, and marketplace economics. State the direction of improvement and the analysis unit for each important metric. **Hint:** A metric tree can help connect the customer problem to operational drivers without treating every correlated measure as a success criterion. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 3: Estimate the Intervention's Causal Effect Design a randomized evaluation if one is feasible. Specify the randomization unit, control condition, exposure logging, analysis population, and comparison. Discuss interference, power, novelty effects, and heterogeneous effects. Then explain how you would estimate impact if randomization is not possible. **Hint:** Match the assignment unit to how the intervention is delivered, and make the identifying assumptions of any observational design testable where possible. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 4: Decide Whether to Launch Suppose the evaluation has finished. Describe a decision framework for launching, iterating, gathering more data, or stopping. Explain how you would handle a statistically significant but tiny improvement, a promising average with harm in one segment, and an inconclusive result. **Hint:** Separate evidence quality, effect size, risk, and reversibility rather than reducing the decision to a p-value threshold. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### What a Strong Answer Covers ```premium-lock What a Strong Answer Covers ``` ### Follow-up Questions 1. Complaints fall after launch, but refunds and delivery times do not change. How would you investigate whether the product worked or merely changed reporting behavior? 2. The intervention is assigned to couriers who serve both treatment and control orders. What bias can this create, and how would you redesign the experiment? 3. The average effect is positive, but long-distance orders become worse. What additional analysis and launch policy would you recommend? 4. Only one market can receive the intervention initially. Which quasi-experimental design would you consider, and what pre-period evidence would you require? 5. How would you calculate the minimum detectable effect and choose an experiment duration when cold-food reports are rare?