Measure Causal Impact of a Self-Selected App Redesign
A mobile app ships a redesigned UI as a new version. Users opt in by upgrading, so a standard randomized A/B test is not possible. Early adopters may differ from non-adopters.
Constraints & Assumptions
-
Treat upgrade as self-selected and staggered over time.
-
Define the causal estimand, such as ATT for adopters.
-
Construct comparable treatment and control groups using pre-upgrade data.
-
Validate assumptions with balance checks, pre-trends, and robustness tests.
Clarifying Questions to Ask
-
What outcome should the redesign affect: engagement, retention, conversion, revenue, or satisfaction?
-
Is adoption voluntary, forced by app store update, or staggered by device/platform?
-
Do users have multiple devices or accounts?
-
Are there concurrent launches, marketing campaigns, or platform changes?
What a Strong Answer Covers
-
Potential-outcomes framing with treatment timing, post-upgrade exposure, and ATT or event-time treatment effect.
-
Threats from self-selection: engagement, device, OS, geography, user tenure, network, and update behavior differences.
-
Comparable groups using propensity-score matching/weighting, exact or coarsened matching, entropy balancing, or doubly robust methods.
-
Covariates beyond past engagement: device/OS, app version eligibility, geography, language, tenure, acquisition channel, notifications, network quality, prior crashes, subscription status, and usage mix.
-
Difference-in-differences or staggered-adoption event study with user and time fixed effects, not-yet-treated controls, and dynamic treatment effects.
-
Validation: covariate balance, common support, pre-trend checks, placebo dates, sensitivity to unobserved confounding, cohort-specific effects, and robustness to alternative windows.
-
Caveat that no observational method fully replaces randomization if key confounders are unobserved.
Follow-up Questions
-
Why is simple pre/post for adopters biased?
-
What makes a good control user?
-
How would you handle users who never upgrade?
-
What if pre-trends are not parallel?