Design a small-sample launch experiment in Europe

Q: Design a small-sample launch experiment in Europe

This question evaluates a data scientist's competencies in experimental design, causal inference, estimand specification, power and MDE calculations for heavy-tailed and overdispersed outcomes, interference handling, compliance/IV considerations, and sequential monitoring in field experiments.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Launch Test Design: Early-Access EU Businesses

Context

You have an early-access pool of 1,200 EU businesses for a new chat subscription offering. Chat volumes are heavy-tailed. You expect 20–30% initial subscription take-up among those offered early access.

You need to estimate the causal impact of the launch on:

Subscription revenue per business.
Resolved-Within-24h Rate (share of chats resolved within 24 hours).

There are interference risks because some customers may interact with multiple businesses. You must choose and justify one design among: a cluster randomized controlled trial (business-level or region-level), a stepped-wedge rollout, or a quasi-experiment.

Task

Design the launch test and provide a full analysis plan, including:

Choice and justification of design (cluster RCT at business-level or region-level, stepped-wedge, or quasi-experiment).
Unit of randomization and stratification variables (country, size, baseline volume).
Handling heavy-tailed outcomes and overdispersion in power/MDE calculations.
Interference risks due to shared customers across businesses.
Compliance and adoption treatment (ITT vs TOT) and IV approach.
Sequential monitoring and alpha spending.
A pre-registered primary analysis plan: estimands, modeling choices, covariate adjustment, heterogeneity (region/vertical), missing data rules.
A decision rule to roll out, iterate, or stop.

Design a small-sample launch experiment in Europe

Quick Overview

Launch Test Design: Early-Access EU Businesses

Context

Task

Solution

Comments (0)