Build a late-delivery risk model

Quick Overview

This question evaluates competencies in production-ready predictive modeling, including target definition, feature engineering, temporal train/validation splitting to avoid leakage, model evaluation and calibration, monitoring and ramp strategies, and translating probabilistic outputs into business-impacting decisions for a Data Scientist.

Build a late-delivery risk model

Company: DoorDash

Role: Data Scientist

Category: Machine Learning

Difficulty: hard

Interview Round: Onsite

You’re given an anonymized DoorDash dataset at order-creation time and asked to predict late delivery risk (late = actual_dropoff_time > quoted_dropoff_time). 1) Define the target precisely and propose a time-based train/validation/test split that avoids leakage from future information (include exact cut dates). 2) Enumerate at least 10 high-signal, production-safe features available at order creation (e.g., store historical on-time rate by hour-of-week, dasher supply-demand index in zone, restaurant prep-time quantiles, distance and traffic, weather, surge/boost, customer lateness tolerance proxy). 3) Identify at least 5 leakage hazards and how you’ll eliminate them (e.g., features derived from post-pickup events, future average wait, features computed with non-causal windows). 4) Choose evaluation metrics for ranking and calibration (e.g., AUROC, AUPRC, Brier, calibration slope), justify thresholds for operational actions, and quantify business impact using a cost matrix. 5) Describe an online ramp: shadow mode -> treatment gating -> A/B test with guardrails; how you’ll monitor drift and recalibrate (e.g., Platt/Isotonic, periodic time-split retraining, population shift alerts). 6) Explain how you’ll handle cold-start restaurants/cities and seasonality (hierarchical pooling, entity embeddings, time-of-week effects).

Quick Answer: This question evaluates competencies in production-ready predictive modeling, including target definition, feature engineering, temporal train/validation splitting to avoid leakage, model evaluation and calibration, monitoring and ramp strategies, and translating probabilistic outputs into business-impacting decisions for a Data Scientist.

|Home/Machine Learning/DoorDash
DoorDash logo
DoorDash
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteMachine Learning
11
0

Predict Late Delivery Risk at Order Creation

Context

You are given an anonymized dataset of marketplace orders with timestamps, store/customer/market attributes, estimated quoted drop-off times (ETA shown at order creation), and realized actual drop-off times. The task is to build a production-ready model that predicts the probability an order will be delivered late when the order is created.

Late is defined by: actual_dropoff_time > quoted_dropoff_time.

Tasks

  1. Target and Time Split
  • Precisely define the binary target label using only information available at order creation (e.g., whether to use the initial quote vs. updated quotes).
  • Propose a time-based train/validation/test split that avoids future-leakage and include exact cut dates.
  1. Features
  • Enumerate at least 10 high-signal, production-safe features that are available at order creation (examples: store historical on-time rate by hour-of-week, dasher supply-demand index, prep-time quantiles, distance/traffic, weather, surge/boost, customer tolerance proxy).
  1. Leakage Hazards
  • Identify at least 5 ways leakage could occur and how to eliminate each (e.g., post-pickup events, future averages, non-causal windows).
  1. Evaluation and Business Impact
  • Choose evaluation metrics for both ranking and calibration (e.g., AUROC, AUPRC, Brier score, calibration slope/intercept).
  • Propose decision thresholds for operational actions and quantify expected business impact using a cost matrix.
  1. Online Ramp and Monitoring
  • Describe an online ramp from shadow mode to controlled rollout, including guardrails.
  • Explain how you will monitor drift and recalibrate (e.g., Platt/Isotonic, periodic time-split retraining, population shift alerts).
  1. Cold Start and Seasonality
  • Explain how you will handle new restaurants/cities and seasonality (e.g., hierarchical pooling, entity embeddings, time-of-week effects).
Loading comments...