Design an Online Experiment
Company: Waymo
Role: Software Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: Onsite
You are asked to design a statistically sound experiment to evaluate whether a new ride-dispatch or scheduling policy improves product performance.
Design the experiment end to end. Your answer should cover:
- the hypothesis and success criteria
- primary metrics and guardrail metrics
- unit of randomization
- treatment assignment strategy
- sample size or experiment duration
- how to handle interference, seasonality, and novelty effects
- the statistical analysis plan
- how you would decide whether to launch
Assume this is a real production system where changes can affect rider wait time, cancellations, and overall service quality.
Quick Answer: This question evaluates experimental design and causal inference skills for production ML systems, including metric definition, randomization and treatment assignment, statistical analysis, and operational concerns like interference and seasonality.