Design a Controlled TV-Advertising Experiment for Sign-Ups
Quick Overview
This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design a Controlled TV-Advertising Experiment for Sign-Ups states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Design a Controlled TV-Advertising Experiment for Sign-Ups
Company: Spokeo
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Hiring-manager question on user-acquisition strategy
##### Question
Design a controlled TV-advertising experiment to increase website sign-ups: state the hypothesis, randomization method, KPIs, required sample size, test length, and success criteria.
##### Hints
Think about geo holdouts, time-based rollout, and lift measurement.
Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design a Controlled TV-Advertising Experiment for Sign-Ups states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Design a Controlled TV-Advertising Experiment for Sign-Ups
Controlled TV Advertising Experiment: Incremental Website Sign-ups
Context
You are asked to design a controlled experiment to quantify the incremental impact of TV ads on website sign-ups. TV here includes linear TV and/or CTV that can be purchased at a geographic level (e.g., DMA). You have historical web sign-ups by geo and time and can avoid national TV during the test window.
Task
Design the experiment and specify:
Hypothesis (H0/H1).
Randomization method and experimental unit (e.g., geo holdouts) and how you’ll select/stratify markets.
Rollout design and timeline (including time-based ON/OFF if used) and how you’ll handle carryover/adstock.
KPIs (primary, secondary, guardrails) and how you’ll measure lift.
Required sample size, MDE, power, and test length; show how you’d compute it and provide a concrete numeric example.
Success criteria (statistical and business/ROI).
Key risks and mitigations (e.g., spillover across geos, concurrent channels, seasonality).
Hint: Consider geo holdouts, time-based rollout, and lift measurement.
Constraints & Assumptions
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask Guidance
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.
What a Strong Answer Covers Guidance
A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions Guidance
What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?