Switchback vs. Geo Randomization for Experiments in a Delivery Marketplace

A food-delivery marketplace connects customers, merchants and couriers. In an experiment-design interview you are asked, in detail, how you would run an experiment on a change that acts on the shared marketplace in a city, and in particular whether you would use a switchback design or geographic (geo) randomization. The report says only that this round probed the choice between switchback and geo randomization and many experiment-design details; it does not say which change was being tested. To make the details concrete, this version uses two example changes: - **Change A:** a new algorithm for assigning orders to couriers. It acts on the live state of the market, and customers and couriers cannot easily tell which version is running. - **Change B:** a new courier pay structure. Couriers learn about it and adjust when and where they work over days or weeks. ### Clarifying Questions - What is the primary decision metric (delivery time, order volume, courier earnings, or a marketplace efficiency measure), and which metrics are guardrails? - Can the change be switched on and off within a city on a schedule, and how quickly does the market settle after a switch? - How many cities or delivery zones are available, how similar are they, and do couriers work across zone boundaries? - What is the smallest effect worth detecting, and how long can the experiment run? ### Part 1 — Choose the design Explain why randomizing individual customers, orders or couriers would give a biased answer for these changes. Then say whether you would test Change A and Change B with a switchback design or with geo randomization, and why. ```hint Follow the shared courier If treated orders are assigned more efficiently, ask what happens to the couriers left over for control orders in the same city at the same moment. ``` #### What This Part Should Cover - The interference mechanism and the direction of bias it creates - What each design randomizes and which assumptions it relies on - A justified choice for each change, based on how fast its effects appear and how long they persist ### Part 2 — Switchback details Design the switchback experiment: the randomization unit, the length of each time window, how to handle carryover between windows, the randomization schedule, the duration, and the analysis, including how you would compute the standard error. ```hint How long is a window Think about how long orders created under one arm keep affecting the market after a switch, and what that implies for the first part of the next window. ``` #### What This Part Should Cover - The unit (region and time window) and the trade-off in window length - Carryover handling and a schedule balanced over time of day and day of week - An analysis whose variance reflects the number of windows and their correlation, not the number of orders ### Part 3 — Geo randomization details Design the geo experiment: which areas are randomized and how they are assigned, the pre-period, the duration, how you handle spillover between neighboring areas, and the analysis and power with a limited number of areas. ```hint Few units, big differences With a limited number of areas that differ a lot from one another, a simple random split can leave the arms unbalanced. Think about what you already know about each area before the test starts. ``` #### What This Part Should Cover - Assignment that balances the arms using pre-period data - A duration long enough for behavior to adjust, and handling of spillover across boundaries - An analysis and a power calculation at the area level ### What a Strong Answer Covers - A clear diagnosis of interference in a shared marketplace and of the bias in individual-level randomization - A decision rule linking a change's characteristics to the design - Concrete parameters for each design, with the reasoning behind them - Standard errors that match the randomization unit, and checks for carryover, spillover and imbalance - Metrics on every side of the marketplace, with guardrails ### Follow-up Questions - How would you test whether your switchback windows are long enough? - With few comparable areas, how would you obtain a trustworthy p-value or confidence interval for the geo test? - A switchback test of Change A shows faster deliveries, but after launch city-level delivery times do not improve. What could explain it? - How would you measure a change that has both an immediate effect and a slow effect on courier behavior?