How would you evaluate adding video ads?

Quick Overview

This question evaluates a data scientist's competencies in experimental design, causal inference, metrics selection, and product monetization analytics within the Analytics & Experimentation domain, focusing on measuring the impact of video ads on revenue, engagement, retention, and lifetime value.

How would you evaluate adding video ads?

Company: Amazon

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

You are a data scientist for a free-to-play mobile game. The product team wants to introduce **video ads** (e.g., rewarded videos and/or interstitial video ads). Design an evaluation plan to determine whether adding video ads is a good idea. Your answer should cover: 1) **Success metrics** and **guardrail metrics** (include tradeoffs: short-term revenue vs. retention/LTV vs. engagement). 2) How you would design an **experiment** (unit of randomization, eligibility, ad frequency/capping, ramp-up, stratification). 3) How you would estimate **incremental impact** on revenue and user experience (e.g., ad revenue offset by changes in IAP purchases, session length, churn). 4) Key threats to validity (novelty effects, selection bias if ads are opt-in, interference/network effects, bots/ad fraud) and how you would mitigate them. 5) How you would decide ship / don’t ship, including how you’d think about **power/MDE** and long-term effects.

Quick Answer: This question evaluates a data scientist's competencies in experimental design, causal inference, metrics selection, and product monetization analytics within the Analytics & Experimentation domain, focusing on measuring the impact of video ads on revenue, engagement, retention, and lifetime value.

|Home/Analytics & Experimentation/Amazon
Amazon logo
Amazon
Nov 4, 2025, 12:00 AM
mediumData ScientistTechnical ScreenAnalytics & Experimentation
6
0

You are a data scientist for a free-to-play mobile game. The product team wants to introduce video ads (e.g., rewarded videos and/or interstitial video ads).

Design an evaluation plan to determine whether adding video ads is a good idea.

Your answer should cover:

  1. Success metrics and guardrail metrics (include tradeoffs: short-term revenue vs. retention/LTV vs. engagement).
  2. How you would design an experiment (unit of randomization, eligibility, ad frequency/capping, ramp-up, stratification).
  3. How you would estimate incremental impact on revenue and user experience (e.g., ad revenue offset by changes in IAP purchases, session length, churn).
  4. Key threats to validity (novelty effects, selection bias if ads are opt-in, interference/network effects, bots/ad fraud) and how you would mitigate them.
  5. How you would decide ship / don’t ship, including how you’d think about power/MDE and long-term effects.
Loading comments...