Design a small-sample launch experiment in Europe

Quick Overview

This question evaluates a data scientist's competencies in experimental design, causal inference, estimand specification, power and MDE calculations for heavy-tailed and overdispersed outcomes, interference handling, compliance/IV considerations, and sequential monitoring in field experiments.

Design a small-sample launch experiment in Europe

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

You have 1,200 EU businesses in an early-access pool, heavy-tailed chat volumes, and expected 20–30% initial subscription take-up. Design a launch test to estimate impact on (1) subscription revenue per business and (2) Resolved-Within-24h Rate. Choose and justify between a cluster RCT (business-level or region-level), a stepped-wedge rollout, or a quasi-experiment. Detail: unit of randomization, stratification variables (country, size, baseline volume), sample size/power and Minimum Detectable Effect under overdispersion, interference risks (shared customers across businesses), compliance and adoption handling (ITT vs TOT with IV), sequential monitoring/alpha spending, and a pre-registered primary analysis. Provide the full analysis plan (estimands, modeling choices, covariate adjustment, heterogeneity by region/vertical, missing data rules), and a decision rule to roll out, iterate, or stop.

Quick Answer: This question evaluates a data scientist's competencies in experimental design, causal inference, estimand specification, power and MDE calculations for heavy-tailed and overdispersed outcomes, interference handling, compliance/IV considerations, and sequential monitoring in field experiments.

|Home/Analytics & Experimentation/Meta
Meta logo
Meta
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteAnalytics & Experimentation
3
0

Launch Test Design: Early-Access EU Businesses

Context

You have an early-access pool of 1,200 EU businesses for a new chat subscription offering. Chat volumes are heavy-tailed. You expect 20–30% initial subscription take-up among those offered early access.

You need to estimate the causal impact of the launch on:

  1. Subscription revenue per business.
  2. Resolved-Within-24h Rate (share of chats resolved within 24 hours).

There are interference risks because some customers may interact with multiple businesses. You must choose and justify one design among: a cluster randomized controlled trial (business-level or region-level), a stepped-wedge rollout, or a quasi-experiment.

Task

Design the launch test and provide a full analysis plan, including:

  • Choice and justification of design (cluster RCT at business-level or region-level, stepped-wedge, or quasi-experiment).
  • Unit of randomization and stratification variables (country, size, baseline volume).
  • Handling heavy-tailed outcomes and overdispersion in power/MDE calculations.
  • Interference risks due to shared customers across businesses.
  • Compliance and adoption treatment (ITT vs TOT) and IV approach.
  • Sequential monitoring and alpha spending.
  • A pre-registered primary analysis plan: estimands, modeling choices, covariate adjustment, heterogeneity (region/vertical), missing data rules.
  • A decision rule to roll out, iterate, or stop.
Loading comments...