Design station experiment with interference and rush-hour spillovers
Overview
This question evaluates a data scientist's competency in experimental design and causal inference under interference and non-stationarity, covering skills such as randomization unit and period selection, KPI and guardrail specification, synthetic control construction, variance-reduction techniques, and diagnostics for spillovers and temporal effects. It is commonly asked in the Analytics & Experimentation domain to assess the ability to design robust, real-world experiments and is primarily a practical application that also requires strong conceptual understanding of bias, carryover, seasonality, shocks, and validation methods.
Experiment Design Under Interference for an In‑Station Ordering Pilot
Context (Completed)
You are evaluating two competing in‑station ordering features (A vs B) at NYC’s Penn Station. The setting has:
- Strong periodicity (weekday AM/PM rush hours).
- Users may share devices or networks (station Wi‑Fi, kiosks).
- Spatial and temporal spillovers (signage, staff behavior, queuing externalities).
- Potential shocks (service disruptions, events) and non‑stationarity over time.
Goal: Determine which feature is better while handling interference and non‑stationarity.
Task
Choose and justify an experimental design under interference—station‑level switchback (A/B by time) versus geo‑split versus synthetic control—and specify:
- Randomization unit, period length, washout, and guardrails to limit contamination.
- Primary KPIs and guardrail metrics (e.g., conversion, throughput/minute, queue abandonment, AOV, CSAT) and a decision rule, including MDE and power assumptions.
- How to handle shocks (service disruptions), trends, and non‑stationarity.
- How to construct a synthetic control (donor pool, pre‑period fit, placebo tests) and how to combine it with switchback results.
- Diagnostics for carryover, novelty, and day‑of‑week effects.
- Variance reduction methods (pairing, CUPED) and how to estimate heterogeneous lift by time block to confidently select the winning feature.
Solution
Login required