Measure Causal Impact of Self-Selected App Redesign
Company: LinkedIn
Role: Data Scientist
Category: Statistics & Math
Difficulty: hard
Interview Round: Onsite
##### Scenario
A mobile-app redesign is shipped as a new version; users opt-in by upgrading, so a standard A/B test is not possible.
##### Question
How would you measure the causal impact of the redesign when users self-select into the new version? Describe the causal-inference framework you would use, how you would construct comparable treatment/control groups, which features you would match or weight on besides past engagement, and how you would validate your assumptions.
##### Hints
Explain propensity-score matching/weighting, covariate selection, balance checks, difference-in-differences or other robustness tests.
Quick Answer: LinkedIn causal inference prompt on measuring a self-selected app redesign, covering potential outcomes, ATT, propensity matching or weighting, staggered difference-in-differences, event studies, covariates, balance, and pre-trend validation.
Measure Causal Impact of Self-Selected App Redesign
LinkedIn
Jul 12, 2025, 6:59 PM
hardData ScientistOnsiteStatistics & Math
73
0
Measure Causal Impact of a Self-Selected App Redesign
A mobile app ships a redesigned UI as a new version. Users opt in by upgrading, so a standard randomized A/B test is not possible. Early adopters may differ from non-adopters.
Constraints & Assumptions
Treat upgrade as self-selected and staggered over time.
Define the causal estimand, such as ATT for adopters.
Construct comparable treatment and control groups using pre-upgrade data.
Validate assumptions with balance checks, pre-trends, and robustness tests.
Clarifying Questions to Ask Guidance
What outcome should the redesign affect: engagement, retention, conversion, revenue, or satisfaction?
Is adoption voluntary, forced by app store update, or staggered by device/platform?
Do users have multiple devices or accounts?
Are there concurrent launches, marketing campaigns, or platform changes?
What a Strong Answer Covers Guidance
Potential-outcomes framing with treatment timing, post-upgrade exposure, and ATT or event-time treatment effect.
Threats from self-selection: engagement, device, OS, geography, user tenure, network, and update behavior differences.
Comparable groups using propensity-score matching/weighting, exact or coarsened matching, entropy balancing, or doubly robust methods.
Covariates beyond past engagement: device/OS, app version eligibility, geography, language, tenure, acquisition channel, notifications, network quality, prior crashes, subscription status, and usage mix.
Difference-in-differences or staggered-adoption event study with user and time fixed effects, not-yet-treated controls, and dynamic treatment effects.
Validation: covariate balance, common support, pre-trend checks, placebo dates, sensitivity to unobserved confounding, cohort-specific effects, and robustness to alternative windows.
Caveat that no observational method fully replaces randomization if key confounders are unobserved.