How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a easy difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at PayPal.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at PayPal during technical interviews.

Design metrics and experiment for donation feature

Quick Overview

A PayPal data science onsite case asking you to design metrics and a randomized experiment for an optional checkout donation feature on a food-ordering platform. Covers goal and hypothesis framing, a primary/diagnostic/guardrail metric hierarchy with precise definitions, user-level experiment design with power and variance-reduction, confounder/marketplace-bias handling, and a launch decision framework.

Question

Uber Eats is considering a new feature: when a user places an order, they can optionally add a donation (to the merchant or a merchant-selected cause) during checkout. You are the data scientist who owns the evaluation. Produce a concise plan that an analytics/DS team could execute.

Goal and hypotheses. What is the main product goal of this feature? State a primary hypothesis and 2-4 secondary hypotheses, including at least one plausible negative effect.
Metrics selection. Define your primary (success) metric(s) , diagnostic metrics (to understand mechanisms), and guardrail metrics (to ensure no harm). Be explicit about definitions: unit of analysis, numerator/denominator, time window, and where in the funnel each metric is measured.
Experiment design. Describe an experiment plan, including: experiment/randomization unit (user/order/merchant), eligibility and experiment population, treatment arm(s) and control, duration and power/MDE considerations, and how you would handle repeated orders, interference/spillover, novelty effects, and heterogeneous treatment effects (e.g., by merchant type or user frequency).
Risks and confounders. List the major risks, biases, and marketplace effects (e.g., selection bias, cannibalization, instrumentation bugs, equilibrium effects) that could mislead conclusions, and how you would address each.
Decision framework. Spell out the launch / no-launch criteria, and how you would weigh tradeoffs (e.g., conversion down but retention up).

Quick Overview

Question

Goal and hypotheses. What is the main product goal of this feature? State a primary hypothesis and 2-4 secondary hypotheses, including at least one plausible negative effect.
Metrics selection. Define your primary (success) metric(s) , diagnostic metrics (to understand mechanisms), and guardrail metrics (to ensure no harm). Be explicit about definitions: unit of analysis, numerator/denominator, time window, and where in the funnel each metric is measured.
Experiment design. Describe an experiment plan, including: experiment/randomization unit (user/order/merchant), eligibility and experiment population, treatment arm(s) and control, duration and power/MDE considerations, and how you would handle repeated orders, interference/spillover, novelty effects, and heterogeneous treatment effects (e.g., by merchant type or user frequency).
Risks and confounders. List the major risks, biases, and marketplace effects (e.g., selection bias, cannibalization, instrumentation bugs, equilibrium effects) that could mislead conclusions, and how you would address each.
Decision framework. Spell out the launch / no-launch criteria, and how you would weigh tradeoffs (e.g., conversion down but retention up).

Design metrics and experiment for donation feature

Quick Overview

Question

Solution

Submit Your Answer to Earn 20XP

Design metrics and experiment for donation feature

Quick Overview

Question

Solution

Submit Your Answer to Earn 20XP