Design a 2‑Week Experiment: $100 Credit After ID Verification

You are designing a 2‑week pilot in which new accounts receive a $100 credit after identity verification. The primary KPI is the 30‑day activation rate: the percentage of new accounts that achieve at least$50 net ad spend within 30 days of signup.

Baseline activation rate is 12%. The budget allows up to 10,000 treatment accounts over 14 days. Daily traffic and the mix vary by region and industry.

Tasks

1. Experimental design
   • Choose the randomization unit.
   • Choose stratification/blocking variables.
   • Explain how you will handle day‑of‑week and regional seasonality.
   • Plan for non‑compliance and fraud.
2. Sample size and power
   • Compute the required sample size for 80% power and α = 0.05 to detect a +2.0 percentage‑point lift in activation (12% → 14%).
   • State assumptions and show the calculation method.
3. Analysis plan
   • Define ITT and TOT; specify how you will estimate each.
   • Describe variance reduction (e.g., CUPED using pre‑treatment covariates).
   • Define guardrail monitoring and pre‑set kill‑switch thresholds for: refund rate, support tickets per 1,000 accounts, average first‑30‑day spend, and CAC.
   • Provide a precise decision rule (e.g., one‑sided test with a superiority margin) and multiplicity control.
4. Data quality
   • Describe how you will check event lag, duplicate accounts, and bots.
5. Heterogeneity of treatment effects (HTE)
   • Describe how you will measure HTE across industry, region, and spend propensity, and how you’ll control for multiple comparisons.
6. External validity
   • Explain how you will ensure results generalize beyond the 2‑week window, including novelty and seasonality adjustments.
7. Rollout decision (scenario)
   • Recommend a rollout plan if the observed lift is +1.6pp with a 95% CI of [+0.2pp, +3.0pp] while support tickets increase by 12%.