A/B Test Design: New Credit-Card Acquisition Flow (Revised APR Disclosure + Signup Bonus)

Context

You are launching a new credit-card acquisition flow that changes the APR disclosure and introduces a revised signup bonus. Design an end-to-end A/B test that measures business impact while maintaining risk and regulatory compliance.

Tasks

1. Experimental design
   • Define the unit of randomization, exposure stickiness, and allocation.
   • Describe how you will ensure assignment consistency across channels/devices.
2. Eligibility and exclusions
   • Specify who is eligible and who is excluded (e.g., existing customers, suspected fraud, do-not-solicit, restricted geographies).
3. Metrics
   • Define the primary success metric.
   • Define secondary and guardrail metrics (e.g., application completion, approval rate, activation rate, early delinquency/default or risk score proxies).
   • Provide clear metric definitions and measurement windows.
4. Powering the test
   • Set the minimal detectable effect (MDE) and statistical power.
   • Compute the required sample size and expected test duration given daily traffic assumptions.
5. Bias and interference
   • Address selection bias (e.g., pre-approved lists, underwriting filters).
   • Address cross-channel interference and identity resolution.
6. Decision integrity
   • Describe how to prevent peeking/early stopping and how you will handle sequential looks if needed.
7. Regulatory and compliance constraints
   • Identify relevant constraints (e.g., fair lending) and how the test will comply.
8. Heterogeneous effects
   • Explain how you would analyze heterogeneous treatment effects by segment while controlling false discovery.