Estimate impact of global launch without holdout
Overview
This question evaluates a candidate's competence in causal inference, observational study design, and statistical uncertainty quantification for product analytics, specifically estimating lift after a global launch without a holdout while addressing confounding, interference, and operational data-quality issues.
Causal Lift Plan After a Global Launch Without a Holdout
Background
A new product feature was launched globally on 2025-05-10, with no control or holdout. You need to estimate the causal impact (lift) on:
- Weekly revenue
- 7-day retention
Assume standard product analytics instrumentation and access to geo-, device-, and cohort-level data.
Task
Design a plan to estimate causal lift using at least two independent identification strategies. For each strategy, specify:
(a) Identification assumptions and key threats
(b) Unit of analysis, data needs, and pre-period length
(c) Diagnostics and falsification tests (e.g., placebo dates, negative/positive control outcomes, pre-trend checks)
(d) Uncertainty quantification and 95% CIs (delta method vs bootstrap; clustering level)
(e) How you will bound effects if assumptions partially fail (e.g., robustness/amplification curves, sensitivity to unobserved confounding)
(f) How you will handle interference/spillovers, seasonality/holidays, concurrent marketing, logging/schema changes, backfilled events, selection into exposure, and migration
Also include:
(g) A decision framework to reconcile conflicting estimates and produce a single recommendation (ship, rollback, or iterate), including acceptable risk thresholds.
Solution
Login required