Design a Causal Upgrade Experiment
Company: Google
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
A company releases a new version of its Android app. All Android users receive a popup asking them to install the update, but only some choose to upgrade. You want to determine whether the new version improves the product.
How would you design an experiment or quasi-experiment to estimate the causal effect while removing selection bias from self-selected updaters? In your answer, specify:
- what product change you would first clarify,
- primary success metrics and guardrail metrics,
- unit of randomization and rollout design,
- how to estimate intent-to-treat and, if needed, treatment-on-the-treated effects,
- and what you would do if only observational data were available after a full launch.
Quick Answer: This question evaluates competency in causal inference and experimental design, focusing on handling selection bias, defining intent-to-treat and treatment-on-the-treated estimands, metric specification, and randomized versus quasi-experimental rollout strategies.