Plan an experiment to validate targeting impact
Overview
This question evaluates a candidate's competency in experimental design, causal inference, statistical power and sample-size calculations, contamination and attribution handling, and cost–benefit analysis for targeted marketing experiments.
You produced a ranked list of merchants predicted to adopt Subscription. Design an experiment to validate business impact of targeting them with a sales/email nudge. Constraints: randomize at merchant level among the top 50,000 scored merchants. Primary metric: conversion to first active Subscription within 30 days. Baseline conversion is 2.0% without outreach; minimal detectable uplift is +50% relative (to 3.0%). Costs: 120. (a) Propose an RCT design (control vs. treatment split, stratification by score deciles and country), guardrails (refund rate, chargeback rate), and a staged ramp. (b) Compute the required per-arm sample size for 80% power and two-sided α=0.05 for proportions (show formula and numeric answer). (c) Given the costs and margin, compute the break-even precision threshold for targeting (ignore long-term spillovers). (d) Discuss contamination risks (multi-merchant customers), how to handle merchants already piloting Subscription, and how to attribute conversions when merchants receive multiple touches.