How to evaluate a new homepage feature

Scenario

PayPal plans to launch a new homepage feature (e.g., a new CTA module, personalized content, or a redesigned layout). You are asked to evaluate whether the feature should be shipped.

Tasks

1. Experiment design
   • Propose an A/B test design (unit of randomization, eligibility, duration considerations, assignment consistency, and key validity checks).
2. Metrics
   • Propose:
     • Primary (success) metric(s)
     • Diagnostic metrics (to understand the mechanism)
     • Guardrail metrics (to ensure no harm)
   • Define each metric clearly (numerator/denominator, event definitions, time windows).
3. Sample size
   • Describe how you would determine the required sample size for the test.
   • State assumptions needed (baseline rate/variance, MDE, alpha, power, variance reduction, clustering, etc.).
4. Increase power
   • List practical ways to increase statistical power (without compromising validity), and explain tradeoffs.

Assume you can instrument user events on the homepage and downstream funnel events (login, add payment method, send money, checkout, etc.).