Measure a Speaker's Impact Using Propensity Score Matching
A company releases an optional smart speaker. Some users purchase and link the speaker to their account, while others do not. Users self-select into ownership, and you cannot run a forced A/B test. You want to measure the causal impact of owning the speaker on engagement.
Constraints & Assumptions
-
Assume user-level panel data on engagement, speaker activation timing, rich pre-treatment covariates, marketing exposure, devices, geography, and staggered adoption.
-
Self-selection is a major concern.
-
The preferred design should compare treated users with comparable non-treated or not-yet-treated users.
-
Include assumptions, diagnostics, and sensitivity analysis.
Clarifying Questions to Ask
-
What engagement outcome matters: streaming hours, sessions, purchases, retention, or device usage?
-
Is treatment defined by purchase, activation, linking, or first use?
-
What pre-treatment history and marketing exposure data are available?
-
Are there never-adopters and not-yet-adopters throughout the study period?
Part 1 - Identification Strategy
How would you measure the causal impact of owning the speaker on engagement?
What This Part Should Cover
-
Propensity score matching or weighting, difference-in-differences with staggered adoption, event study, or combined matched DiD.
-
Estimand such as ATT.
-
Assumptions: conditional exchangeability, overlap, no anticipation, and parallel trends if using DiD.
Part 2 - Construct Control Group
How would you construct the control group?
What This Part Should Cover
-
Propensity score model using pre-treatment engagement, trends, demographics, geography, devices, marketing exposure, and prior product usage.
-
Matching, weighting, calipers, common support, and balance diagnostics.
-
Avoiding post-treatment variables.
Part 3 - Estimate and Validate
How would you estimate impact and validate the design?
What This Part Should Cover
-
Outcome model, matched or weighted comparison, DiD/event-study regression, robust standard errors, placebo tests, pre-trend checks, and sensitivity to unobserved confounding.
-
Heterogeneity and duration effects.
What a Strong Answer Covers
A strong answer recognizes self-selection, constructs comparable controls with pre-treatment information, validates balance and pre-trends, and reports uncertainty and sensitivity rather than treating ownership as randomized.
Follow-up Questions
-
What if adopters were already increasing engagement before purchase?
-
How would you handle users influenced by a marketing campaign?
-
What if overlap between adopters and non-adopters is poor?