Prove new allocation outperforms manual baseline

Read the full interview experience this question came from →

Quick Overview

This question evaluates skills in experimental design, causal inference, statistical power/MDE calculations, metric definition, and operational governance for large‑scale randomized trials in logistics.

Prove new allocation outperforms manual baseline

Company: Amazon

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

Propose a rigorous plan to prove an automated package-allocation system outperforms the manual baseline. Address interference and spillovers by choosing a randomization unit (e.g., station-hour switchback or courier-level cluster randomization) and justify it. Define primary/guardrail metrics (on-time rate, SLA breaches, courier overtime, customer contacts, fairness Gini), pre-specify analysis (intention-to-treat, CUPED/covariate adjustment, heterogeneity by zip density), and a power/MDE calculation (baseline on-time 92%, 100k orders/day, intra-cluster correlation assumptions). Include ramp/rollback criteria, duration, and a diff-in-diff fallback if perfect randomization isn’t feasible. Explain how you’ll prevent gaming and contamination (inventory locking, shadow assignments), and how you’ll conclude superiority with statistical and practical significance (e.g., uplift ≥0.6 pp with non-inferior guardrails).

Overview: This question evaluates skills in experimental design, causal inference, statistical power/MDE calculations, metric definition, and operational governance for large‑scale randomized trials in logistics.

Read the full Amazon Data Scientist interview experience this question came from

|Home/Amazon/Analytics & Experimentation
Amazon logo
Amazon
Oct 13, 2025
hardData ScientistOnsiteAnalytics & Experimentation
7
0

Prove an Automated Package-Allocation System Outperforms Manual Baseline

Context

You work in a large last‑mile logistics network evaluating a new automated package‑allocation (dispatch) system versus the current manual process. Daily volume is ~100,000 orders. The baseline on‑time delivery rate is 92%.

Task

Design a rigorous experiment and analysis plan that:

  1. Addresses interference and spillovers
    • Propose and justify a randomization unit (e.g., station‑hour switchback or courier‑level cluster randomization).
    • Explain operational controls to limit contamination.
  2. Defines metrics and decision criteria
    • Primary: on‑time delivery rate.
    • Guardrails: SLA breaches, courier overtime, customer contacts, fairness (Gini of work allocation).
    • Set superiority and non‑inferiority thresholds (e.g., uplift ≥ 0.6 pp for primary, non‑inferior guardrails).
  3. Pre‑specifies the analysis
    • Intention‑to‑treat (ITT) as primary; note any per‑protocol sensitivity.
    • Variance reduction (CUPED/covariate adjustment).
    • Heterogeneity by zip‑code density (urban vs suburban/rural).
    • Inference approach and handling of clustering.
  4. Includes power/MDE calculations
    • Use baseline on‑time = 92%, 100k orders/day.
    • State intra‑cluster correlation assumptions.
    • Provide duration estimates to detect a 0.6 pp uplift.
  5. Covers operations and governance
    • Ramp plan, monitoring, and rollback criteria.
    • Diff‑in‑diff fallback if perfect randomization isn’t feasible.
    • Anti‑gaming/contamination controls (e.g., inventory locking, shadow assignments).

Browse More Questions

More Analytics & Experimentation•More Amazon•More Data Scientist•Amazon Data Scientist•Amazon Analytics & Experimentation•Data Scientist Analytics & Experimentation