Plan an experiment to validate targeting impact

Quick Overview

This question evaluates a candidate's competency in experimental design, causal inference, statistical power and sample-size calculations, contamination and attribution handling, and cost–benefit analysis for targeted marketing experiments.

Plan an experiment to validate targeting impact

Company: Stripe

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Take-home Project

You produced a ranked list of merchants predicted to adopt Subscription. Design an experiment to validate business impact of targeting them with a sales/email nudge. Constraints: randomize at merchant level among the top 50,000 scored merchants. Primary metric: conversion to first active Subscription within 30 days. Baseline conversion is 2.0% without outreach; minimal detectable uplift is +50% relative (to 3.0%). Costs: $5 outreach cost per merchant; expected 90-day gross margin from a converted merchant is $120. (a) Propose an RCT design (control vs. treatment split, stratification by score deciles and country), guardrails (refund rate, chargeback rate), and a staged ramp. (b) Compute the required per-arm sample size for 80% power and two-sided α=0.05 for proportions (show formula and numeric answer). (c) Given the costs and margin, compute the break-even precision threshold for targeting (ignore long-term spillovers). (d) Discuss contamination risks (multi-merchant customers), how to handle merchants already piloting Subscription, and how to attribute conversions when merchants receive multiple touches.

Quick Answer: This question evaluates a candidate's competency in experimental design, causal inference, statistical power and sample-size calculations, contamination and attribution handling, and cost–benefit analysis for targeted marketing experiments.

|Home/Analytics & Experimentation/Stripe
Stripe logo
Stripe
Oct 13, 2025, 9:49 PM
mediumData ScientistTake-home ProjectAnalytics & Experimentation
2
0

You produced a ranked list of merchants predicted to adopt Subscription. Design an experiment to validate business impact of targeting them with a sales/email nudge. Constraints: randomize at merchant level among the top 50,000 scored merchants. Primary metric: conversion to first active Subscription within 30 days. Baseline conversion is 2.0% without outreach; minimal detectable uplift is +50% relative (to 3.0%). Costs: 5outreachcostpermerchant;expected90daygrossmarginfromaconvertedmerchantis5 outreach cost per merchant; expected 90-day gross margin from a converted merchant is 120. (a) Propose an RCT design (control vs. treatment split, stratification by score deciles and country), guardrails (refund rate, chargeback rate), and a staged ramp. (b) Compute the required per-arm sample size for 80% power and two-sided α=0.05 for proportions (show formula and numeric answer). (c) Given the costs and margin, compute the break-even precision threshold for targeting (ignore long-term spillovers). (d) Discuss contamination risks (multi-merchant customers), how to handle merchants already piloting Subscription, and how to attribute conversions when merchants receive multiple touches.

Loading comments...