Design a Controlled TV-Advertising Experiment for Sign-Ups
Overview
This question evaluates a data scientist's skills in experimental design, causal impact measurement for marketing channels, statistical power and sample-size calculations, KPI definition, and dealing with temporal and spatial confounders within the Analytics & Experimentation domain.
Spokeo
Aug 4, 2025, 10:55 AM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Controlled TV Advertising Experiment: Incremental Website Sign-ups
Context
You are asked to design a controlled experiment to quantify the incremental impact of TV ads on website sign-ups. TV here includes linear TV and/or CTV that can be purchased at a geographic level (e.g., DMA). You have historical web sign-ups by geo and time and can avoid national TV during the test window.
Task
Design the experiment and specify:
- Hypothesis (H0/H1).
- Randomization method and experimental unit (e.g., geo holdouts) and how you’ll select/stratify markets.
- Rollout design and timeline (including time-based ON/OFF if used) and how you’ll handle carryover/adstock.
- KPIs (primary, secondary, guardrails) and how you’ll measure lift.
- Required sample size, MDE, power, and test length; show how you’d compute it and provide a concrete numeric example.
- Success criteria (statistical and business/ROI).
- Key risks and mitigations (e.g., spillover across geos, concurrent channels, seasonality).
Hint: Consider geo holdouts, time-based rollout, and lift measurement.
Solution
Login required
Show