Plan an experiment to validate targeting impact
Company: Stripe
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: Medium
Interview Round: Take-home Project
You produced a ranked list of merchants predicted to adopt Subscription. Design an experiment to validate business impact of targeting them with a sales/email nudge. Constraints: randomize at merchant level among the top 50,000 scored merchants. Primary metric: conversion to first active Subscription within 30 days. Baseline conversion is 2.0% without outreach; minimal detectable uplift is +50% relative (to 3.0%). Costs: $5 outreach cost per merchant; expected 90-day gross margin from a converted merchant is $120. (a) Propose an RCT design (control vs. treatment split, stratification by score deciles and country), guardrails (refund rate, chargeback rate), and a staged ramp. (b) Compute the required per-arm sample size for 80% power and two-sided α=0.05 for proportions (show formula and numeric answer). (c) Given the costs and margin, compute the break-even precision threshold for targeting (ignore long-term spillovers). (d) Discuss contamination risks (multi-merchant customers), how to handle merchants already piloting Subscription, and how to attribute conversions when merchants receive multiple touches.
Quick Answer: This question evaluates a candidate's competency in experimental design, causal inference, statistical power and sample-size calculations, contamination and attribution handling, and cost–benefit analysis for targeted marketing experiments.