Design an Unbiased Upgrade Experiment
Company: Google
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
The Google app releases a new Android version. Every Android user sees a pop-up encouraging installation, but installation is voluntary: some users upgrade and some do not.
How would you determine whether the new version is better than the old one?
Your answer should:
- define the product change and the causal estimand
- choose primary and guardrail metrics
- explain why comparing upgraders versus non-upgraders is biased
- design a randomized experiment or encouragement design that addresses selection bias
- describe how you would analyze intent-to-treat versus treatment-on-the-treated effects
- discuss what you would do if the rollout already happened and only observational data remain
Quick Answer: This question evaluates understanding of causal inference, experiment design, and metric selection in product analytics, including defining causal estimands and distinguishing intent-to-treat versus treatment-on-the-treated effects.