Design a small-sample launch experiment in Europe
Overview
This question evaluates a data scientist's competencies in experimental design, causal inference, estimand specification, power and MDE calculations for heavy-tailed and overdispersed outcomes, interference handling, compliance/IV considerations, and sequential monitoring in field experiments.
Launch Test Design: Early-Access EU Businesses
Context
You have an early-access pool of 1,200 EU businesses for a new chat subscription offering. Chat volumes are heavy-tailed. You expect 20–30% initial subscription take-up among those offered early access.
You need to estimate the causal impact of the launch on:
- Subscription revenue per business.
- Resolved-Within-24h Rate (share of chats resolved within 24 hours).
There are interference risks because some customers may interact with multiple businesses. You must choose and justify one design among: a cluster randomized controlled trial (business-level or region-level), a stepped-wedge rollout, or a quasi-experiment.
Task
Design the launch test and provide a full analysis plan, including:
- Choice and justification of design (cluster RCT at business-level or region-level, stepped-wedge, or quasi-experiment).
- Unit of randomization and stratification variables (country, size, baseline volume).
- Handling heavy-tailed outcomes and overdispersion in power/MDE calculations.
- Interference risks due to shared customers across businesses.
- Compliance and adoption treatment (ITT vs TOT) and IV approach.
- Sequential monitoring and alpha spending.
- A pre-registered primary analysis plan: estimands, modeling choices, covariate adjustment, heterogeneity (region/vertical), missing data rules.
- A decision rule to roll out, iterate, or stop.
Solution
Login required