Prove conversion ads value via incrementality
Overview
This question evaluates a candidate's competency in causal inference and experimentation design for advertising measurement, covering metrics and attribution, statistical power and sample-size calculations, interference handling, and business-level impact estimation.
TikTok
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
Loading...
Measuring Incremental Lift of Conversion-Optimized Ads
Context
An e-commerce advertiser is running conversion-optimized ads and requests rigorous proof that the ads cause incremental conversions (lift). You are asked to design and analyze a trustworthy measurement plan that isolates causality under real-world constraints (auctions, pacing, cross-device, view-throughs).
Task
Propose a defensible, end-to-end plan that covers:
-
Design choice and assumptions
- Choose among: user-level RCT holdouts, geo-experiments with matched markets, or PSA/ghost-bid approaches.
- Justify the choice, clearly state assumptions, and discuss spillover/interference risks.
-
Metrics and attribution windows
- Define the primary metric (absolute and/or relative lift in conversion rate) and the attribution windows.
- Address view-through bias, cross-device identity, and post-view latency.
-
Power and sample size
- Perform sample-size/power calculations to detect a 2% relative lift from a 3% baseline conversion rate (CR) at α = 0.05.
- Show formulas, parameters, and assumptions.
-
Analysis plan and guardrails
- Outline the analysis (e.g., difference-in-differences, CUPED, pre-trend diagnostics), interference handling, budget/pacing constraints, and how to guard against p-hacking.
-
Business readout
- Compute incremental ROAS with confidence intervals.
- Explain how to present results and actionability to the advertiser, including sensitivity analyses and repeatability criteria.
Solution
Login required
Show