How to evaluate adding video ads in a game
Company: Amazon
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
## Case study: Evaluate adding video ads to a mobile game
You are the Data Scientist for a free-to-play mobile game. The product team wants to add **video ads** (could be **rewarded** and/or **interstitial**) to increase monetization.
### Task
Describe how you would **evaluate** whether adding video ads is a good idea, and how you would decide whether to roll it out.
Your answer should cover:
1. **Goal & decision**
- What is the decision to be made (ship / not ship / ship with constraints)?
- What does “success” mean?
2. **Metrics (with tradeoffs)**
- Propose **primary** metric(s) and explain why.
- Propose **diagnostic** metric(s) to understand mechanisms.
- Propose **guardrail** metric(s) to ensure you don’t harm the game.
3. **Experiment / identification strategy**
- How would you design an A/B test (unit of randomization, duration, ramp, segmentation/stratification)?
- What confounders or biases are likely (e.g., seasonality, payer vs non-payer mix, novelty effects), and how would you address them?
4. **Analysis plan**
- How would you estimate the **incremental** impact on total revenue and long-term value (LTV)?
- How would you handle heavy-tailed monetization metrics and delayed impact?
5. **Rollout & monitoring**
- What rollout plan would you recommend (e.g., frequency caps, targeting rules)?
- What would you monitor post-launch?
State any assumptions you need to make.
Quick Answer: This question evaluates skills in product analytics, experimentation design, causal inference and monetization modeling for free-to-play mobile games, including metrics selection, A/B testing, and lifetime value estimation.
