Design and Analyze A/B Test for Cashback Program

Q: Design and Analyze A/B Test for Cashback Program

This question evaluates experimental design, causal inference, statistical analysis, instrumentation, and product analytics competencies for running and interpreting A/B tests on promotional checkout features.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

A/B Test Design: Checkout Cashback Program (PayPal)

Scenario

PayPal plans to launch a checkout cashback program (e.g., "Get 1–5% back when you pay with PayPal"). The goal is to evaluate whether offering cashback at checkout improves key business outcomes while remaining cost-effective and safe.

Task

Design, run, and analyze an A/B test to evaluate the cashback program.

Please cover

Hypothesis and success criteria.
Experiment design:
- Population and eligibility
- Unit of randomization and variants
- Exposure and assignment rules
- Sample size and test duration
Metrics:
- Primary metric(s)
- Guardrail/safety metrics
- Secondary/diagnostic metrics
Data needed and instrumentation.
Analysis plan and statistical methods.
Segmentation and heterogeneity of effects.
Risks, biases, and mitigations (e.g., fraud, interference, novelty).

You may assume the feature is shown at checkout to eligible users and credits cashback after successful payment.

Design and Analyze A/B Test for Cashback Program

A/B Test Design: Checkout Cashback Program (PayPal)

Scenario

Task

Please cover

Solution

Comments (0)