Design Experiment to Evaluate New Video-Ad Effectiveness
Quick Overview
Evaluates experiment design for a new video-ad format in an auction-based consumer app. Strong answers choose randomization units that manage marketplace interference, define advertiser, revenue, and user guardrail metrics, handle power and sequential testing, and plan safe rollout.
Design Experiment to Evaluate New Video-Ad Effectiveness
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
The company plans to launch a new video-ad format and needs an experimental framework to evaluate its effectiveness.
##### Question
Design an experiment to measure the impact of the new video ads. Include unit of randomization, primary/secondary metrics, sample-size/power considerations, and success criteria. If the primary metric shows no statistically significant improvement, what follow-up analyses or alternative actions would you take?
##### Hints
Cover A/B vs. multivariate, geographic splits, sequential testing, metric deep-dive, heterogeneous treatment effects, experiment extensions.
Quick Answer: Evaluates experiment design for a new video-ad format in an auction-based consumer app. Strong answers choose randomization units that manage marketplace interference, define advertiser, revenue, and user guardrail metrics, handle power and sequential testing, and plan safe rollout.
Design an Experiment to Evaluate New Video-Ad Effectiveness
A large consumer app is considering a new video-ad format with changes to UI, creative rendering, or interaction. The goal is to estimate causal impact on advertiser effectiveness and revenue while protecting user experience.
Constraints & Assumptions
Ads are allocated through an auction, so experiment design may create marketplace interference.
Include randomization unit, metrics, power, success criteria, and rollout.
Protect user experience, latency, integrity, and revenue guardrails.
Discuss how the design changes if the first test is inconclusive.
Clarifying Questions to Ask Guidance
What changed in the video ad: rendering, length, placement, interaction, or targeting?
Is the goal advertiser lift, platform revenue, user experience, or all three?
Can the format be randomized post-auction, or does it affect auction ranking?
Are advertiser budgets and pacing affected by the treatment?
Part 1 - Randomization Design
What unit of randomization would you choose, and why?
What This Part Should Cover Guidance
Impression-level, user-level, advertiser-level, geo-level, or post-auction ghost/shadow options.
Auction interference and marketplace spillovers.
A phased approach if needed: isolated rendering test, then marketplace-level validation.
Part 2 - Metrics
What primary, secondary, and guardrail metrics would you define?
What This Part Should Cover Guidance
Advertiser outcomes such as view-through rate, clicks, conversions, cost per action, ROAS, and lift.
Platform outcomes such as revenue, eCPM, fill, auction health, and advertiser retention.
User guardrails such as retention, session time, hides, reports, load time, latency, and complaints.
Part 3 - Power and Analysis
How would you handle sample size, power, variance reduction, clustering, and sequential testing?
What This Part Should Cover Guidance
MDE, traffic allocation, duration, heavy-tailed outcomes, clustered or robust standard errors, CUPED, and pre-specified stopping rules.
Sample-ratio mismatch and instrumentation checks.
Part 4 - Success Criteria and Rollout
What launch criteria and rollout plan would you use?
What This Part Should Cover Guidance
Predefined thresholds for primary metric lift and guardrails.
Ramped rollout, monitoring, rollback plan, and follow-up tests.
What to do if results are inconclusive or mixed.
What a Strong Answer Covers Guidance
A strong answer respects ad-marketplace interference, chooses a defensible randomization design, measures advertiser, platform, and user outcomes, and uses staged rollout with clear guardrails.
Follow-up Questions Guidance
What if advertisers in treatment spend budget faster?
How would you test creative quality separately from ad placement?