Design an Experiment to Evaluate New Video-Ad Effectiveness
A large consumer app is considering a new video-ad format with changes to UI, creative rendering, or interaction. The goal is to estimate causal impact on advertiser effectiveness and revenue while protecting user experience.
Constraints & Assumptions
-
Ads are allocated through an auction, so experiment design may create marketplace interference.
-
Include randomization unit, metrics, power, success criteria, and rollout.
-
Protect user experience, latency, integrity, and revenue guardrails.
-
Discuss how the design changes if the first test is inconclusive.
Clarifying Questions to Ask
-
What changed in the video ad: rendering, length, placement, interaction, or targeting?
-
Is the goal advertiser lift, platform revenue, user experience, or all three?
-
Can the format be randomized post-auction, or does it affect auction ranking?
-
Are advertiser budgets and pacing affected by the treatment?
Part 1 - Randomization Design
What unit of randomization would you choose, and why?
What This Part Should Cover
-
Impression-level, user-level, advertiser-level, geo-level, or post-auction ghost/shadow options.
-
Auction interference and marketplace spillovers.
-
A phased approach if needed: isolated rendering test, then marketplace-level validation.
Part 2 - Metrics
What primary, secondary, and guardrail metrics would you define?
What This Part Should Cover
-
Advertiser outcomes such as view-through rate, clicks, conversions, cost per action, ROAS, and lift.
-
Platform outcomes such as revenue, eCPM, fill, auction health, and advertiser retention.
-
User guardrails such as retention, session time, hides, reports, load time, latency, and complaints.
Part 3 - Power and Analysis
How would you handle sample size, power, variance reduction, clustering, and sequential testing?
What This Part Should Cover
-
MDE, traffic allocation, duration, heavy-tailed outcomes, clustered or robust standard errors, CUPED, and pre-specified stopping rules.
-
Sample-ratio mismatch and instrumentation checks.
Part 4 - Success Criteria and Rollout
What launch criteria and rollout plan would you use?
What This Part Should Cover
-
Predefined thresholds for primary metric lift and guardrails.
-
Ramped rollout, monitoring, rollback plan, and follow-up tests.
-
What to do if results are inconclusive or mixed.
What a Strong Answer Covers
A strong answer respects ad-marketplace interference, chooses a defensible randomization design, measures advertiser, platform, and user outcomes, and uses staged rollout with clear guardrails.
Follow-up Questions
-
What if advertisers in treatment spend budget faster?
-
How would you test creative quality separately from ad placement?
-
What if revenue rises but user retention falls?