##### Question PayPal is planning to launch a new **homepage feature** (for example, a new CTA module, personalized content, or a redesigned layout) and wants to decide whether to ship it. Design an experiment to evaluate the feature. Assume this is a consumer-facing homepage experience for logged-in users (justify explicitly if you change scope), and that you can instrument homepage events and downstream funnel events (login, add payment method, send money, checkout, etc.). Your answer should cover: 1. **Objective and hypothesis** — State the experiment objective and the null vs alternative hypothesis. 2. **Experiment design** — Propose an A/B test design: unit of randomization (and why it is appropriate), eligibility, assignment consistency, duration considerations, and key validity checks. 3. **Metrics** — Propose and clearly define (numerator/denominator, event definition, time window): - one **primary (success) metric**, - **secondary / diagnostic funnel metrics**, - **guardrail metrics**. 4. **Metric tradeoffs** — Discuss the tradeoffs among candidate metrics. For example, the feature might increase homepage engagement but also slow the page, distract users from core payment flows, or change fraud risk. 5. **Sample size** — Describe how you would determine the required sample size, including the inputs needed (baseline rate/variance, minimum detectable effect, significance level, power, traffic allocation, clustering, etc.). 6. **Increase power** — List practical ways to increase statistical power if traffic is limited, without compromising validity, and explain the tradeoffs. 7. **Pitfalls** — Call out common pitfalls such as novelty effects, seasonality, user heterogeneity (Simpson's paradox), logging errors, and interference across users or devices. 8. **Ship decision** — Define a decision rule for whether to ship.

A PayPal data science technical screen on designing an A/B test to evaluate a new homepage feature. It covers experiment design and randomization, primary/secondary/guardrail metric selection with tradeoffs, sample-size and power estimation, and pitfalls like novelty effects and interference. The solution emphasizes value-aligned metrics (payment completion over CTR) and fraud guardrails specific to a payments product.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a easy difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at PayPal.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at PayPal during technical interviews.

How to evaluate a new homepage feature | PayPal Interview Question

Question

PayPal is planning to launch a new homepage feature (for example, a new CTA module, personalized content, or a redesigned layout) and wants to decide whether to ship it. Design an experiment to evaluate the feature.

Assume this is a consumer-facing homepage experience for logged-in users (justify explicitly if you change scope), and that you can instrument homepage events and downstream funnel events (login, add payment method, send money, checkout, etc.).

Your answer should cover:

Objective and hypothesis — State the experiment objective and the null vs alternative hypothesis.
Experiment design — Propose an A/B test design: unit of randomization (and why it is appropriate), eligibility, assignment consistency, duration considerations, and key validity checks.
Metrics — Propose and clearly define (numerator/denominator, event definition, time window):
- one primary (success) metric ,
- secondary / diagnostic funnel metrics ,
- guardrail metrics .
Metric tradeoffs — Discuss the tradeoffs among candidate metrics. For example, the feature might increase homepage engagement but also slow the page, distract users from core payment flows, or change fraud risk.
Sample size — Describe how you would determine the required sample size, including the inputs needed (baseline rate/variance, minimum detectable effect, significance level, power, traffic allocation, clustering, etc.).
Increase power — List practical ways to increase statistical power if traffic is limited, without compromising validity, and explain the tradeoffs.
Pitfalls — Call out common pitfalls such as novelty effects, seasonality, user heterogeneity (Simpson's paradox), logging errors, and interference across users or devices.
Ship decision — Define a decision rule for whether to ship.

Question

Your answer should cover:

Objective and hypothesis — State the experiment objective and the null vs alternative hypothesis.
Experiment design — Propose an A/B test design: unit of randomization (and why it is appropriate), eligibility, assignment consistency, duration considerations, and key validity checks.
Metrics — Propose and clearly define (numerator/denominator, event definition, time window):
- one primary (success) metric ,
- secondary / diagnostic funnel metrics ,
- guardrail metrics .
Metric tradeoffs — Discuss the tradeoffs among candidate metrics. For example, the feature might increase homepage engagement but also slow the page, distract users from core payment flows, or change fraud risk.
Sample size — Describe how you would determine the required sample size, including the inputs needed (baseline rate/variance, minimum detectable effect, significance level, power, traffic allocation, clustering, etc.).
Increase power — List practical ways to increase statistical power if traffic is limited, without compromising validity, and explain the tradeoffs.
Pitfalls — Call out common pitfalls such as novelty effects, seasonality, user heterogeneity (Simpson's paradox), logging errors, and interference across users or devices.
Ship decision — Define a decision rule for whether to ship.

How to evaluate a new homepage feature

Quick Overview

Question

Solution

Submit Your Answer to Earn 20XP

How to evaluate a new homepage feature

Quick Overview

Question

Solution

Submit Your Answer to Earn 20XP