How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at PayPal.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at PayPal during technical interviews.

Design A/B Test to Measure PayPal Cashback Value

Quick Overview

This question evaluates an interviewee's ability to design and analyze a randomized experiment to quantify incremental value for a retail partner, covering experiment design, metric taxonomy, power/sample-size calculations, statistical testing, and variance-reduction techniques.

Scenario

PayPal plans to offer a targeted cashback incentive for purchases at Walmart. You need to design an A/B test that convincingly demonstrates the value this cashback creates for Walmart (not just for PayPal), while respecting practical constraints of time and data.

Task

Structure an end-to-end experiment to measure the value for Walmart, including:

Experiment design
- Unit of randomization, eligibility, treatment/control definition, stratification, exposure/compliance, instrumentation, and interference controls.
Metric taxonomy
- Define primary, secondary, and guard-rail metrics that reflect value to Walmart.
Power analysis and sample-size math
- Show how you would set the minimum detectable effect (MDE), estimate variance, compute sample size, and adjust for clustering or variance reduction.
Statistical testing plan
- Explain p-value interpretation, multiple testing control, and whether/how you’d use sequential testing.
If results miss the MDE
- Outline steps to salvage learning and decision-making.
If power is insufficient and the timeline cannot be extended
- Propose alternatives such as variance reduction (e.g., CUPED/CUBED), pre-post designs, or geo/synthetic-control approaches that allow a credible read within the fixed time.

Hints

Include metric definitions and formulas.
Show example numbers for sample size calculations.
Discuss sequential testing and variance reduction (e.g., CUPED/CUBED) and mitigation plans for data/operational risks.

Quick Overview

Task

Structure an end-to-end experiment to measure the value for Walmart, including:

Experiment design

Unit of randomization, eligibility, treatment/control definition, stratification, exposure/compliance, instrumentation, and interference controls.

Metric taxonomy

Define primary, secondary, and guard-rail metrics that reflect value to Walmart.

Power analysis and sample-size math

Show how you would set the minimum detectable effect (MDE), estimate variance, compute sample size, and adjust for clustering or variance reduction.

Statistical testing plan

Explain p-value interpretation, multiple testing control, and whether/how you’d use sequential testing.

If results miss the MDE

Outline steps to salvage learning and decision-making.

If power is insufficient and the timeline cannot be extended

Propose alternatives such as variance reduction (e.g., CUPED/CUBED), pre-post designs, or geo/synthetic-control approaches that allow a credible read within the fixed time.

Design A/B Test to Measure PayPal Cashback Value

Quick Overview

Scenario

Task

Hints

Solution

Comments (0)

Design A/B Test to Measure PayPal Cashback Value

Quick Overview

Scenario

Task

Hints

Solution

Comments (0)