This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design A/B Test to Measure PayPal Cashback Value states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
##### Scenario
Design an A/B test to prove PayPal cashback delivers value to Walmart.
##### Question
How would you structure the experiment to measure value for Walmart? Define primary, secondary and guard-rail metrics. Describe your power analysis approach and explain p-value interpretation. If results miss the minimum detectable effect, what steps would you take? Power is insufficient but timeline cannot be extended—what alternatives (e.g., CUBED) can help?
##### Hints
Show end-to-end experimental design: metric taxonomy, sample-size math, sequential testing or CUPED/CUBED fixes, and mitigation plans.
Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design A/B Test to Measure PayPal Cashback Value states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
PayPal plans to offer a targeted cashback incentive for purchases at Walmart. You need to design an A/B test that convincingly demonstrates the value this cashback creates for Walmart (not just for PayPal), while respecting practical constraints of time and data.
Task
Structure an end-to-end experiment to measure the value for Walmart, including:
Experiment design
Unit of randomization, eligibility, treatment/control definition, stratification, exposure/compliance, instrumentation, and interference controls.
Metric taxonomy
Define primary, secondary, and guard-rail metrics that reflect value to Walmart.
Power analysis and sample-size math
Show how you would set the minimum detectable effect (MDE), estimate variance, compute sample size, and adjust for clustering or variance reduction.
Statistical testing plan
Explain p-value interpretation, multiple testing control, and whether/how you’d use sequential testing.
If results miss the MDE
Outline steps to salvage learning and decision-making.
If power is insufficient and the timeline cannot be extended
Propose alternatives such as variance reduction (e.g., CUPED/CUBED), pre-post designs, or geo/synthetic-control approaches that allow a credible read within the fixed time.
Hints
Include metric definitions and formulas.
Show example numbers for sample size calculations.
Discuss sequential testing and variance reduction (e.g., CUPED/CUBED) and mitigation plans for data/operational risks.
Clarifying Questions to Ask Guidance
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.
What a Strong Answer Covers Guidance
A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions Guidance
What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?