Platform for Running Product Experiments at Google: Scope, Users and Metrics
Company: Google
Role: Product Manager
Category: Product Design & Strategy
Difficulty: medium
Interview Round: Other
You are a product manager at Google, asked to design a platform for running experiments. Decide what kind of experimentation it supports and for whom, what it must do, and how you would know it is succeeding.
```hint Resolve the scope, then follow one experiment
"Experimentation" can describe several different products. Once you pick one, trace a single experiment from idea to decision and ask where it gets slow or produces results nobody should trust.
```
### Clarifying Questions
- Is this an internal platform for Google's product teams to run controlled experiments such as A/B tests, a program that lets users opt in to try early features, or a service for outside developers?
- Which products should it support first: Search, YouTube, Android apps, Ads, or all of them?
- Does it replace tools that teams already use, so migration matters, or does it start from scratch?
- Is the main goal faster launch decisions, more trustworthy decisions, or revenue?
### What a Strong Answer Covers
- Resolving the ambiguous scope, then naming who sets up experiments, who reads them and who makes the call
- Pain points across the experiment life cycle, from design and traffic allocation to analysis and the launch decision
- A prioritized feature set that balances self-service speed with statistical rigor
- Success metrics for the platform itself, such as decision speed, trust and adoption, with guardrails like serving overhead and user harm
- Risks specific to experiments at Google's scale: overlapping tests, users influencing each other, early peeking and conflicting metric definitions
### Follow-up Questions
- Two teams want to test changes on the same search results page at the same time. How does the platform keep both results valid?
- YouTube creators and viewers affect each other. When would splitting by user give a biased answer, and what would you offer instead?
- How would you stop teams from checking results daily and shipping the first change that looks significant?
Overview: A Google product manager interview question asking for the design of a platform to run experiments. It tests scoping an ambiguous prompt, understanding who sets up and reads A/B tests, balancing self-service speed with statistical rigor, and defining success metrics for the platform itself.