Predict Late Delivery Risk at Order Creation

Context

You are given an anonymized dataset of marketplace orders with timestamps, store/customer/market attributes, estimated quoted drop-off times (ETA shown at order creation), and realized actual drop-off times. The task is to build a production-ready model that predicts the probability an order will be delivered late when the order is created.

Late is defined by: actual_dropoff_time > quoted_dropoff_time.

Tasks

1. Target and Time Split

• Precisely define the binary target label using only information available at order creation (e.g., whether to use the initial quote vs. updated quotes).
• Propose a time-based train/validation/test split that avoids future-leakage and include exact cut dates.

2. Features

• Enumerate at least 10 high-signal, production-safe features that are available at order creation (examples: store historical on-time rate by hour-of-week, dasher supply-demand index, prep-time quantiles, distance/traffic, weather, surge/boost, customer tolerance proxy).

3. Leakage Hazards

• Identify at least 5 ways leakage could occur and how to eliminate each (e.g., post-pickup events, future averages, non-causal windows).

4. Evaluation and Business Impact

• Choose evaluation metrics for both ranking and calibration (e.g., AUROC, AUPRC, Brier score, calibration slope/intercept).
• Propose decision thresholds for operational actions and quantify expected business impact using a cost matrix.

5. Online Ramp and Monitoring

• Describe an online ramp from shadow mode to controlled rollout, including guardrails.
• Explain how you will monitor drift and recalibrate (e.g., Platt/Isotonic, periodic time-split retraining, population shift alerts).

6. Cold Start and Seasonality

• Explain how you will handle new restaurants/cities and seasonality (e.g., hierarchical pooling, entity embeddings, time-of-week effects).