Evaluate a New Homepage Feature
Company: PayPal
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
PayPal is considering launching a new homepage feature. Design an experiment to evaluate whether the feature should be shipped.
Your answer should cover:
1. The experiment objective and hypothesis.
2. The randomization unit and why it is appropriate.
3. A set of success metrics, including:
- one primary metric,
- secondary funnel metrics,
- guardrail metrics.
4. Tradeoffs among possible metrics. For example, the feature might increase homepage engagement but also slow the page, distract users from core payment flows, or affect fraud risk.
5. How to estimate the required sample size, including the inputs needed (baseline rate, minimum detectable effect, significance level, power, variance assumptions, traffic allocation, etc.).
6. Practical ways to increase statistical power if traffic is limited.
7. Common pitfalls such as novelty effects, seasonality, user heterogeneity, logging errors, and interference across users or devices.
Assume this is a consumer-facing homepage experience for logged-in users unless you explicitly justify a different scope.
Quick Answer: This question evaluates a data scientist's skills in experimental design, causal inference, metric selection (primary, secondary, guardrail), statistical power/sample-size estimation, and threat modeling for a consumer-facing homepage feature for logged-in users.