Measure Speaker's Impact Using Propensity Score Matching
Quick Overview
Evaluates causal inference for optional smart speaker ownership using propensity score matching and staggered adoption. Strong answers address self-selection, construct matched or weighted controls, validate balance and pre-trends, estimate ATT, and run placebo and sensitivity analyses.
Measure Speaker's Impact Using Propensity Score Matching
Company: Roku
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
The company releases an optional smart speaker and wants to understand its effect on user engagement, but users self-select into purchasing.
##### Question
Without being able to run a forced A/B test, how would you measure the causal impact of owning the speaker on engagement? Describe your preferred design and why.
##### Hints
Discuss quasi-experiments: difference-in-differences, propensity-score matching, instrumental variables, pre-post checks.
Quick Answer: Evaluates causal inference for optional smart speaker ownership using propensity score matching and staggered adoption. Strong answers address self-selection, construct matched or weighted controls, validate balance and pre-trends, estimate ATT, and run placebo and sensitivity analyses.
Measure a Speaker's Impact Using Propensity Score Matching
A company releases an optional smart speaker. Some users purchase and link the speaker to their account, while others do not. Users self-select into ownership, and you cannot run a forced A/B test. You want to measure the causal impact of owning the speaker on engagement.
Constraints & Assumptions
Assume user-level panel data on engagement, speaker activation timing, rich pre-treatment covariates, marketing exposure, devices, geography, and staggered adoption.
Self-selection is a major concern.
The preferred design should compare treated users with comparable non-treated or not-yet-treated users.
Include assumptions, diagnostics, and sensitivity analysis.
Clarifying Questions to Ask Guidance
What engagement outcome matters: streaming hours, sessions, purchases, retention, or device usage?
Is treatment defined by purchase, activation, linking, or first use?
What pre-treatment history and marketing exposure data are available?
Are there never-adopters and not-yet-adopters throughout the study period?
Part 1 - Identification Strategy
How would you measure the causal impact of owning the speaker on engagement?
What This Part Should Cover Guidance
Propensity score matching or weighting, difference-in-differences with staggered adoption, event study, or combined matched DiD.
Estimand such as ATT.
Assumptions: conditional exchangeability, overlap, no anticipation, and parallel trends if using DiD.
Part 2 - Construct Control Group
How would you construct the control group?
What This Part Should Cover Guidance
Propensity score model using pre-treatment engagement, trends, demographics, geography, devices, marketing exposure, and prior product usage.
Matching, weighting, calipers, common support, and balance diagnostics.
Avoiding post-treatment variables.
Part 3 - Estimate and Validate
How would you estimate impact and validate the design?
What This Part Should Cover Guidance
Outcome model, matched or weighted comparison, DiD/event-study regression, robust standard errors, placebo tests, pre-trend checks, and sensitivity to unobserved confounding.
Heterogeneity and duration effects.
What a Strong Answer Covers Guidance
A strong answer recognizes self-selection, constructs comparable controls with pre-treatment information, validates balance and pre-trends, and reports uncertainty and sensitivity rather than treating ownership as randomized.
Follow-up Questions Guidance
What if adopters were already increasing engagement before purchase?
How would you handle users influenced by a marketing campaign?
What if overlap between adopters and non-adopters is poor?