Design Experiment to Evaluate New Video-Ad Effectiveness
Overview
This question evaluates competency in experimental design and causal inference, including randomized assignment, metric definition, power and sample-size reasoning, interference management in auctioned ad ecosystems, and interpretation of sequential tests and rollout criteria.
Meta
Jul 12, 2025, 6:59 PM
Data Scientist
Technical Screen
Analytics & Experimentation
77
0
Experiment Design: Measuring the Impact of a New Video‑Ad Format
Context
You are designing an experiment for a large consumer app that runs an ad auction. A new video‑ad format (UI/creative/interaction changes) is being considered. The goal is to estimate its causal impact on advertiser effectiveness and revenue while protecting user experience, accounting for potential interference in the ad marketplace.
Task
Design an experiment to evaluate the new format. Specify:
- Unit of randomization (and why), including options (A/B vs multivariate, user vs impression vs geo) and how to manage auction/interference.
- Primary and secondary metrics (with definitions and guardrails).
- Sample size and power considerations (including variance reduction, clustering, and sequential testing).
- Success criteria and a rollout plan.
- If the primary metric shows no statistically significant improvement, list follow‑up analyses and alternative actions.
Solution
Login required
Show