Multi-Arm A/B Test: Comparing Cost-Per-Conversion Across Channels

Scenario

You need to compare four new acquisition channels—YouTube ads, Google Search ads, Facebook ads, and Direct Mail—to choose the most cost-efficient option for driving conversions given a fixed budget.

Task

Design a rigorous multi-arm A/B test to evaluate cost-per-conversion efficiency across these channels.

Address the following:

1. Primary metric
   • What exactly will you optimize? Define the metric precisely (including incrementality vs. attribution, unit of analysis, and conversion window).
2. Experimental design
   • Randomization scheme (units, arms, control), avoiding cross-channel contamination, frequency caps, and deduped conversions.
3. Hypotheses and statistical test
   • State the null and alternative hypotheses, and specify the appropriate global and pairwise tests.
4. Sample size, budget split, and duration
   • How will you determine these given desired power and minimum detectable effect (MDE)? Include how per-user costs differ by channel.
5. Post-hoc / follow-up analyses
   • What analyses will you run after the main test (e.g., multiple comparisons, heterogeneity, creative, response curves)?

Hints: Discuss cost-per-conversion metric, multi-arm design, power/alpha, ANOVA vs. pairwise tests, budget allocation, assumptions, and demographic/creative differences.