# Reduce Cold-Food Incidents with Metrics and an Insulated-Bag Experiment A delivery marketplace wants to reduce the support costs, credits, and refunds caused by customers receiving cold food. The primary metric is flat for four weeks and rises in weeks five through eight. The team proposes giving couriers insulated bags. ### Constraints & Assumptions - Complaints and credits are imperfect proxies for food temperature. - Couriers may not consistently use the assigned bag. - Bag assignment can affect multiple deliveries by the same courier. - Geo-level tests have fewer independent units and may face spillovers. ### Clarifying Questions to Ask - Is the objective customer experience, direct cost, or both? - How is a cold-food incident identified and when is it considered complete? - Can bags be randomized by courier, zone, or shift? ### Part 1 — Metrics and diagnosis Define a primary metric and guardrails, then investigate the delayed increase in weeks five through eight. #### What This Part Should Cover - Incident cost and rate definitions - Numerator, denominator, lag, and composition checks - Courier, restaurant, distance, weather, and logging slices ### Part 2 — Experiment Design the bag test, account for non-use and novelty, and compare courier randomization with geo-level testing. #### What This Part Should Cover - Assignment, exposure, compliance, and intent-to-treat - Power and interference at the chosen unit - Novelty, learning, durability, and safety guardrails ### Part 3 — Borderline p-value The estimated effect is favorable but the predeclared test returns p = 0.055. Explain what you would do. #### What This Part Should Cover - Effect size and confidence interval - No post-hoc threshold changes - Decision risk, replication, and additional data under a predeclared rule ### What a Strong Answer Covers - Definitions tied to customer and economic outcomes - A defensible randomization unit - An interpretation that respects the analysis plan ```hint Measure assignment and use separately Keep an intent-to-treat comparison based on bag assignment, then estimate usage and compliance as secondary analyses rather than dropping noncompliant couriers. ``` ### Follow-up Questions - How could you verify bag use without invasive monitoring? - What if bags help long trips but slow short trips?