Design A/B Test for Marketing Campaign Impact Evaluation
Overview
This question evaluates experimental design and causal inference competencies for measuring incremental card acquisitions and cardholder spend, covering metric definition, assignment strategy, guardrail identification, and risk-aware analysis within the Analytics & Experimentation domain.
Scenario
A credit card issuer is testing a marketing campaign that waives the first-year annual fee. You need to design an A/B test to measure incremental card acquisitions and cardholder spend while ensuring customer experience and risk do not degrade.
Assume the campaign can run across web and email to prospect audiences and that you can randomize at the person level consistently across channels. The standard offer (control) charges the first-year annual fee; the variant (treatment) waives it.
Task
Design the experiment and analysis plan. Include:
- Experiment design and assignment.
- Primary outcome metrics and how they are computed.
- Guardrail metrics (with rationale).
- Sample-size calculations for acquisition and spend (state assumptions and show formulas with a small numeric example).
- Post-experiment analysis steps, including variance reduction (e.g., CUPED or stratification).
Hints
- Focus on conversion rate and revenue/spend lift.
- Consider CUPED or stratified randomization to increase power.
- Address heavy-tailed spend distributions and operational/risk guardrails.
Solution
Login required