How would you evaluate adding video ads?
Company: Amazon
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
You are a data scientist for a free-to-play mobile game. The product team wants to introduce **video ads** (e.g., rewarded videos and/or interstitial video ads).
Design an evaluation plan to determine whether adding video ads is a good idea.
Your answer should cover:
1) **Success metrics** and **guardrail metrics** (include tradeoffs: short-term revenue vs. retention/LTV vs. engagement).
2) How you would design an **experiment** (unit of randomization, eligibility, ad frequency/capping, ramp-up, stratification).
3) How you would estimate **incremental impact** on revenue and user experience (e.g., ad revenue offset by changes in IAP purchases, session length, churn).
4) Key threats to validity (novelty effects, selection bias if ads are opt-in, interference/network effects, bots/ad fraud) and how you would mitigate them.
5) How you would decide ship / don’t ship, including how you’d think about **power/MDE** and long-term effects.
Quick Answer: This question evaluates a data scientist's competencies in experimental design, causal inference, metrics selection, and product monetization analytics within the Analytics & Experimentation domain, focusing on measuring the impact of video ads on revenue, engagement, retention, and lifetime value.
