A new product feature was launched globally on 2025-05-10, with no control or holdout. You need to estimate the causal impact (lift) on:
Assume standard product analytics instrumentation and access to geo-, device-, and cohort-level data.
Design a plan to estimate causal lift using at least two independent identification strategies. For each strategy, specify:
(a) Identification assumptions and key threats
(b) Unit of analysis, data needs, and pre-period length
(c) Diagnostics and falsification tests (e.g., placebo dates, negative/positive control outcomes, pre-trend checks)
(d) Uncertainty quantification and 95% CIs (delta method vs bootstrap; clustering level)
(e) How you will bound effects if assumptions partially fail (e.g., robustness/amplification curves, sensitivity to unobserved confounding)
(f) How you will handle interference/spillovers, seasonality/holidays, concurrent marketing, logging/schema changes, backfilled events, selection into exposure, and migration
Also include:
(g) A decision framework to reconcile conflicting estimates and produce a single recommendation (ship, rollback, or iterate), including acceptable risk thresholds.
Login required