Design an experiment with marketplace network effects

Quick Overview

This question evaluates a candidate's competency in causal experiment design for networked two‑sided marketplaces, focusing on interference and spillover management, cluster randomization choices, metric definition, power/sample‑size calculations, bias‑reduction approaches, and diagnostic checks.

Design an experiment with marketplace network effects

Company: Uber

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

Uber plans to launch a new networked product in a two‑sided marketplace (riders and drivers). How would you design a causal experiment that accounts for interference/network effects? Specify: (a) the unit of randomization (user, driver, city, or geo‑cluster) and why; (b) how you’ll limit and measure spillovers (e.g., cluster randomization, geographic lift tests, partial‑interference assumptions, graph cuts/holdouts); (c) primary and guardrail metrics and how you’ll compute them; (d) power/sample‑size calculations under clustering (ICC, design effect) and expected duration; (e) bias‑reduction techniques (e.g., CUPED, pre‑period stratification) and novelty/washout handling; (f) diagnostics you’ll run to detect SUTVA violations and noncompliance; and (g) how the design changes if driver supply is effectively unlimited. Provide a concrete rollout plan and analysis outline.

Quick Answer: This question evaluates a candidate's competency in causal experiment design for networked two‑sided marketplaces, focusing on interference and spillover management, cluster randomization choices, metric definition, power/sample‑size calculations, bias‑reduction approaches, and diagnostic checks.

|Home/Analytics & Experimentation/Uber
Uber logo
Uber
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
6
0

Causal Experiment Design for a Two‑Sided Marketplace with Interference

You are designing a causal experiment for a new networked product in a two‑sided marketplace (riders and drivers) where interference/network effects are likely (e.g., through matching, wait times, or pricing).

Specify the following:

  1. Unit of randomization (user, driver, city, or geo‑cluster) and why.
  2. How you will limit and measure spillovers (e.g., cluster randomization, geographic lift tests, partial‑interference assumptions, graph cuts/holdouts).
  3. Primary and guardrail metrics and how to compute them.
  4. Power/sample‑size calculations under clustering (including ICC and design effect) and expected duration.
  5. Bias‑reduction techniques (e.g., CUPED, pre‑period stratification) and how you’ll handle novelty/washout.
  6. Diagnostics to detect SUTVA violations and noncompliance.
  7. How the design changes if driver supply is effectively unlimited.

Provide a concrete rollout plan and analysis outline.

Loading comments...