Supply Chain Fulfillment ETA Modeling

What's being tested

Interviewers are probing your ability to build, evaluate, and operationalize ETA (Estimated Time of Arrival) models for fulfillment — balancing predictive accuracy, calibrated uncertainty, and business impact. You'll be judged on metric design, label engineering (including censoring and late-arrival handling), segmentation/cohort evaluation, and how you trade off model complexity versus interpretability for downstream uses like SLAs and customer communications.

Core knowledge

Label definition & censoring: define ETA as either time-to-event (seconds/minutes) or absolute timestamp; account for right-censoring when deliveries are in-flight at evaluation cutoff and for missing final-status events.
Time-to-event / survival framing: use survival analysis or hazard models when censoring matters; survival function $S(t)=P(T>t)$ and hazard $h(t)=\frac{f(t)}{S(t)}$ let you produce full distributional ETAs.
Distributional predictions & quantiles: prefer quantile regression or direct interval models to produce p50/p90/p95 estimates; optimize pinball loss for quantiles: $L_{q}(u)=\max(qu, (q-1)u)$ where $u=y-\hat{y}$ .
Point metrics vs business metrics: report MAE and RMSE for point accuracy, but prioritize p90/p95 on SLA cohorts and weighted errors for high-value orders; MAE = $\frac{1}{n}\sum|y_i-\hat{y}_i|$ .
Calibration & reliability: check coverage for predicted intervals (e.g., 95% CI covers ~95% of outcomes); use reliability diagrams and conformal prediction for distributional calibration.
Feature design (signals): pull timestamps and status updates from Postgres/event logs and real-time signals like GPS, courier assignment, historical route durations, weather, and order size. Encode time-to-pickup, time-of-day, weekday, and distance-to-destination as core predictors.
Model families & tradeoffs: tree ensembles (XGBoost, LightGBM) are strong baselines for tabular ETA; survival forests or gradient-boosted survival models handle censoring. Neural sequence models help for spatiotemporal streams but cost latency.
Evaluation windows & leakage: split data by event time (not ingestion time) and avoid using signals that only exist post-label (e.g., final scan). Simulate production latency: only use features available at the prediction time.
Error attribution & segmentation: calculate cohort-level metrics (by region, courier, order size) and delta decomposition — model error = bias + variance + irreducible noise — to prioritize feature collection or local models.
Experimentation & causal inference: when ETA model changes affect downstream behavior (re-routing, customer cancellations), design A/B tests instrumenting treatment exposure; measure both prediction metrics and business KPIs (on-time rate, customer complaints).
Operational concerns (DS scope): specify monitoring metrics: drift on feature distributions, model calibration, p95 SLA misses; propose retraining cadence and backfills, but leave pipeline plumbing to Data Engineering.
Uncertainty communication: for UX, prefer range + confidence (e.g., ETA 45–60 min, p90=60) and explain tradeoff: tighter intervals increase risk of SLA misses.

Worked example — "Predict package ETA for last-mile delivery"

Frame: first ask clarifying Qs — what is the prediction timestamp (order placed, last-mile dispatch?), what SLA or business metric matters (p95 on same-day deliveries?), and what latency constraints exist for serving. Skeleton: (1) define label precisely (time from prediction timestamp to final delivery), handle right-censoring for undelivered events, (2) assemble features available at prediction time (courier status, route distance, real-time GPS if available, historical travel-times by route/time), (3) choose modeling approach — baseline LightGBM for point p50 with separate quantile models for p90/p95 or a survival-gradient-boosted model, (4) evaluation plan — time-based cross-validation, report MAE and p95 miss-rate by cohort, and (5) calibration step — isotonic regression or conformal intervals to ensure coverage. Tradeoff to flag: whether to serve a single global model (faster, less overfit) or per-region models (better local accuracy, more maintenance). Close with: if I had more time I'd instrument an online A/B test to measure behavioral impacts of narrower ETAs on cancellations and monitor p95 drift post-launch.

A second angle — "Detect systemic ETA drift causing p95 misses"

This reframes the problem from prediction to monitoring and root-cause. Start by defining the alerting metric: an increase in p95 ETA error or p95 SLA miss-rate across cohorts. Approach: segment by courier, time-of-day, and route; run significance tests on recent vs baseline error distributions (use bootstrapped CI or sequential testing). For causes, compare feature distribution shifts (e.g., average route speed down by X%), correlate external signals (traffic, weather), and run counterfactual checks: was a model change deployed? The DS role focuses on diagnostics and causal attribution, not fixing the streaming pipeline — propose remediation: retrain on recent data, change feature windows, or fall back to simpler robust baseline. Emphasize validating fixes with pre-post A/B tests to confirm reduced p95 misses.

Common pitfalls

Pitfall: leaking future signals into features — including post-dispatch scans or final courier-reported durations in training gives unrealistically low errors that fail in production.

Pitfall: reporting only point estimates (mean/p50) — stakeholders care about tail behavior; a model with good MAE but poor p95 will still break SLAs.

Pitfall: optimizing global metrics without cohort checks — an improvement in average error can hide regressions on high-value or rural cohorts; always report segmented KPIs and business-weighted objectives.

Connections

This area commonly pivots to adjacent problems: inventory provisioning (how ETA uncertainty affects safety stock), dynamic routing (real-time ETA feeds into route optimization), and experimentation infrastructure (sequential testing and measurement of downstream behavioral effects).