Scenario

PayPal plans to offer a targeted cashback incentive for purchases at Walmart. You need to design an A/B test that convincingly demonstrates the value this cashback creates for Walmart (not just for PayPal), while respecting practical constraints of time and data.

Task

Structure an end-to-end experiment to measure the value for Walmart, including:

1. Experiment design
   • Unit of randomization, eligibility, treatment/control definition, stratification, exposure/compliance, instrumentation, and interference controls.
2. Metric taxonomy
   • Define primary, secondary, and guard-rail metrics that reflect value to Walmart.
3. Power analysis and sample-size math
   • Show how you would set the minimum detectable effect (MDE), estimate variance, compute sample size, and adjust for clustering or variance reduction.
4. Statistical testing plan
   • Explain p-value interpretation, multiple testing control, and whether/how you’d use sequential testing.
5. If results miss the MDE
   • Outline steps to salvage learning and decision-making.
6. If power is insufficient and the timeline cannot be extended
   • Propose alternatives such as variance reduction (e.g., CUPED/CUBED), pre-post designs, or geo/synthetic-control approaches that allow a credible read within the fixed time.

Hints

• Include metric definitions and formulas.
• Show example numbers for sample size calculations.
• Discuss sequential testing and variance reduction (e.g., CUPED/CUBED) and mitigation plans for data/operational risks.