Measure speaker impact without A/B testing

Q: Measure speaker impact without A/B testing

This is a Analytics & Experimentation interview question from Roku for Data Scientist roles. View the full question and solution on PracHub.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Your company launched an optional smart speaker on 2025-01-10. Adoption is voluntary; 6% of existing app users purchase within 60 days, and you cannot force adoption or run an A/B test. Available data: user-level app logs, device graph, CRM attributes, ad exposures, prices, shipments, and regional stockouts. A) Define and justify one primary engagement metric and guardrails (e.g., churn, substitution to other devices). B) Design two complementary identification strategies to estimate the causal effect on engagement at +30 and +180 days: (1) staggered-adoption difference-in-differences/event-study; (2) propensity-score matched IPW/DR or a synthetic control at the user/region level. For each, write the estimand, key assumptions, diagnostics (pre-trends, covariate balance, negative controls), and how you will handle interference (household spillovers) and seasonality. C) Leverage exogenous variation from regional stockouts/shipping delays as an instrument: argue relevance and exclusion, and outline falsification tests. D) Provide a back-of-the-envelope power/MDE given 6% adoption and historical variance: what sample size or horizon is needed to detect a 3% lift?

Measure speaker impact without A/B testing

Comments (0)