Design metrics and experiment for donation feature

Quick Overview

A PayPal data science onsite case asking you to design metrics and a randomized experiment for an optional checkout donation feature on a food-ordering platform. Covers goal and hypothesis framing, a primary/diagnostic/guardrail metric hierarchy with precise definitions, user-level experiment design with power and variance-reduction, confounder/marketplace-bias handling, and a launch decision framework.

Design metrics and experiment for donation feature

Company: PayPal

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: easy

Interview Round: Onsite

##### Question Uber Eats is considering a new feature: when a user places an order, they can optionally **add a donation** (to the merchant or a merchant-selected cause) during checkout. You are the data scientist who owns the evaluation. Produce a concise plan that an analytics/DS team could execute. 1. **Goal and hypotheses.** What is the main product goal of this feature? State a **primary** hypothesis and 2-4 **secondary** hypotheses, including at least one plausible **negative** effect. 2. **Metrics selection.** Define your **primary (success) metric(s)**, **diagnostic metrics** (to understand mechanisms), and **guardrail metrics** (to ensure no harm). Be explicit about definitions: unit of analysis, numerator/denominator, time window, and where in the funnel each metric is measured. 3. **Experiment design.** Describe an experiment plan, including: experiment/randomization unit (user/order/merchant), eligibility and experiment population, treatment arm(s) and control, duration and power/MDE considerations, and how you would handle repeated orders, interference/spillover, novelty effects, and heterogeneous treatment effects (e.g., by merchant type or user frequency). 4. **Risks and confounders.** List the major risks, biases, and marketplace effects (e.g., selection bias, cannibalization, instrumentation bugs, equilibrium effects) that could mislead conclusions, and how you would address each. 5. **Decision framework.** Spell out the launch / no-launch criteria, and how you would weigh tradeoffs (e.g., conversion down but retention up).

Quick Answer: A PayPal data science onsite case asking you to design metrics and a randomized experiment for an optional checkout donation feature on a food-ordering platform. Covers goal and hypothesis framing, a primary/diagnostic/guardrail metric hierarchy with precise definitions, user-level experiment design with power and variance-reduction, confounder/marketplace-bias handling, and a launch decision framework.

|Home/Analytics & Experimentation/PayPal
PayPal logo
PayPal
Dec 16, 2025, 12:00 AM
easyData ScientistOnsiteAnalytics & Experimentation
6
0
Question

Uber Eats is considering a new feature: when a user places an order, they can optionally add a donation (to the merchant or a merchant-selected cause) during checkout. You are the data scientist who owns the evaluation. Produce a concise plan that an analytics/DS team could execute.

  1. Goal and hypotheses. What is the main product goal of this feature? State a primary hypothesis and 2-4 secondary hypotheses, including at least one plausible negative effect.
  2. Metrics selection. Define your primary (success) metric(s) , diagnostic metrics (to understand mechanisms), and guardrail metrics (to ensure no harm). Be explicit about definitions: unit of analysis, numerator/denominator, time window, and where in the funnel each metric is measured.
  3. Experiment design. Describe an experiment plan, including: experiment/randomization unit (user/order/merchant), eligibility and experiment population, treatment arm(s) and control, duration and power/MDE considerations, and how you would handle repeated orders, interference/spillover, novelty effects, and heterogeneous treatment effects (e.g., by merchant type or user frequency).
  4. Risks and confounders. List the major risks, biases, and marketplace effects (e.g., selection bias, cannibalization, instrumentation bugs, equilibrium effects) that could mislead conclusions, and how you would address each.
  5. Decision framework. Spell out the launch / no-launch criteria, and how you would weigh tradeoffs (e.g., conversion down but retention up).
Loading comments...