Choose between A/B and switchback for spillovers
Overview
This question evaluates experimental-design and causal-inference competencies, specifically handling interference and spillovers, defining experimental units and treatment assignment, drawing causal DAGs, identifying bias sources, selecting primary and guardrail metrics, and pre-specifying decision rules.
Airport Driver-Queue Algorithm: Experiment Design and Causal Reasoning
Background
A new driver-queue algorithm is being tested at a single airport with multiple terminals. The algorithm can reassign drivers across terminals, which may influence nearby terminals and the overall airport driver pool.
Task
Decide whether to use a classic user-level A/B test or a switchback/geo-time experiment. Then:
- Define the experimental unit and explain how treatment will be assigned.
- Draw and explain a causal DAG to reason about interference and spillovers across terminals and over time.
- Justify your experimental design choice given potential interference.
- Identify main bias sources (e.g., time trends, weather, event traffic) and propose mitigation strategies (e.g., blocking by hour/day, CUPED, randomization inference).
-
Specify:
- Primary success metric
- Guardrail metrics
- A clear, pre-specified go/no-go decision rule
Assume you can toggle the algorithm on/off at the airport level and collect standard operational metrics (wait times, cancellations, earnings, throughput).
Solution
Login required