Design and Analyze A/B Test for Cashback Program
Overview
This question evaluates experimental design, causal inference, statistical analysis, instrumentation, and product analytics competencies for running and interpreting A/B tests on promotional checkout features.
PayPal
Aug 4, 2025, 10:55 AM
Data Scientist
Onsite
Analytics & Experimentation
0
0
A/B Test Design: Checkout Cashback Program (PayPal)
Scenario
PayPal plans to launch a checkout cashback program (e.g., "Get 1–5% back when you pay with PayPal"). The goal is to evaluate whether offering cashback at checkout improves key business outcomes while remaining cost-effective and safe.
Task
Design, run, and analyze an A/B test to evaluate the cashback program.
Please cover
- Hypothesis and success criteria.
-
Experiment design:
- Population and eligibility
- Unit of randomization and variants
- Exposure and assignment rules
- Sample size and test duration
-
Metrics:
- Primary metric(s)
- Guardrail/safety metrics
- Secondary/diagnostic metrics
- Data needed and instrumentation.
- Analysis plan and statistical methods.
- Segmentation and heterogeneity of effects.
- Risks, biases, and mitigations (e.g., fraud, interference, novelty).
You may assume the feature is shown at checkout to eligible users and credits cashback after successful payment.
Solution
Login required
Show