Design and Analyze A/B Test for Cashback Program

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design and Analyze A/B Test for Cashback Program states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Design and Analyze A/B Test for Cashback Program

Company: PayPal

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Onsite

##### Scenario A/B testing and product analytics discussions ##### Question Walk me through how you would design, run, and analyze an A/B test for a new checkout feature. Case study: PayPal plans to launch a cashback program. How would you evaluate its success, what metrics would you track, and what data would you need? ##### Hints State hypothesis, choose primary & guardrail metrics, randomization, sample-size, test duration, segmentation, analysis plan.

Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design and Analyze A/B Test for Cashback Program states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/PayPal
PayPal logo
PayPal
Aug 4, 2025, 10:55 AM
mediumData ScientistOnsiteAnalytics & Experimentation
4
0

Design and Analyze A/B Test for Cashback Program

A/B Test Design: Checkout Cashback Program (PayPal)

Scenario

PayPal plans to launch a checkout cashback program (e.g., "Get 1–5% back when you pay with PayPal"). The goal is to evaluate whether offering cashback at checkout improves key business outcomes while remaining cost-effective and safe.

Task

Design, run, and analyze an A/B test to evaluate the cashback program.

Please cover

  1. Hypothesis and success criteria.
  2. Experiment design:
    • Population and eligibility
    • Unit of randomization and variants
    • Exposure and assignment rules
    • Sample size and test duration
  3. Metrics:
    • Primary metric(s)
    • Guardrail/safety metrics
    • Secondary/diagnostic metrics
  4. Data needed and instrumentation.
  5. Analysis plan and statistical methods.
  6. Segmentation and heterogeneity of effects.
  7. Risks, biases, and mitigations (e.g., fraud, interference, novelty).

You may assume the feature is shown at checkout to eligible users and credits cashback after successful payment.

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More PayPal•More Data Scientist•PayPal Data Scientist•PayPal Analytics & Experimentation•Data Scientist Analytics & Experimentation