Design A/B Test for Cost-Per-Conversion Efficiency Analysis
Overview
This question evaluates a data scientist's competency in experimental design and metrics engineering for multi-arm A/B tests, including defining cost-per-conversion metrics, randomization and contamination controls, sample size and power calculations, statistical hypothesis testing, and post-hoc heterogeneity analyses.
Multi-Arm A/B Test: Comparing Cost-Per-Conversion Across Channels
Scenario
You need to compare four new acquisition channels—YouTube ads, Google Search ads, Facebook ads, and Direct Mail—to choose the most cost-efficient option for driving conversions given a fixed budget.
Task
Design a rigorous multi-arm A/B test to evaluate cost-per-conversion efficiency across these channels.
Address the following:
-
Primary metric
- What exactly will you optimize? Define the metric precisely (including incrementality vs. attribution, unit of analysis, and conversion window).
-
Experimental design
- Randomization scheme (units, arms, control), avoiding cross-channel contamination, frequency caps, and deduped conversions.
-
Hypotheses and statistical test
- State the null and alternative hypotheses, and specify the appropriate global and pairwise tests.
-
Sample size, budget split, and duration
- How will you determine these given desired power and minimum detectable effect (MDE)? Include how per-user costs differ by channel.
-
Post-hoc / follow-up analyses
- What analyses will you run after the main test (e.g., multiple comparisons, heterogeneity, creative, response curves)?
Hints: Discuss cost-per-conversion metric, multi-arm design, power/alpha, ANOVA vs. pairwise tests, budget allocation, assumptions, and demographic/creative differences.
Solution
Login required