Measure speaker impact without A/B testing

Quick Overview

This question evaluates causal inference, observational study design, metric specification, instrumental variable reasoning, handling of interference and seasonality, and basic power/MDE calculations in a data science analytics context.

Measure speaker impact without A/B testing

Company: Roku

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

Your company launched an optional smart speaker on 2025-01-10. Adoption is voluntary; 6% of existing app users purchase within 60 days, and you cannot force adoption or run an A/B test. Available data: user-level app logs, device graph, CRM attributes, ad exposures, prices, shipments, and regional stockouts. A) Define and justify one primary engagement metric and guardrails (e.g., churn, substitution to other devices). B) Design two complementary identification strategies to estimate the causal effect on engagement at +30 and +180 days: (1) staggered-adoption difference-in-differences/event-study; (2) propensity-score matched IPW/DR or a synthetic control at the user/region level. For each, write the estimand, key assumptions, diagnostics (pre-trends, covariate balance, negative controls), and how you will handle interference (household spillovers) and seasonality. C) Leverage exogenous variation from regional stockouts/shipping delays as an instrument: argue relevance and exclusion, and outline falsification tests. D) Provide a back-of-the-envelope power/MDE given 6% adoption and historical variance: what sample size or horizon is needed to detect a 3% lift?

Quick Answer: This question evaluates causal inference, observational study design, metric specification, instrumental variable reasoning, handling of interference and seasonality, and basic power/MDE calculations in a data science analytics context.

|Home/Analytics & Experimentation/Roku
Roku logo
Roku
Oct 13, 2025, 9:49 PM
mediumData ScientistTechnical ScreenAnalytics & Experimentation
8
0

Your company launched an optional smart speaker on 2025-01-10. Adoption is voluntary; 6% of existing app users purchase within 60 days, and you cannot force adoption or run an A/B test. Available data: user-level app logs, device graph, CRM attributes, ad exposures, prices, shipments, and regional stockouts. A) Define and justify one primary engagement metric and guardrails (e.g., churn, substitution to other devices). B) Design two complementary identification strategies to estimate the causal effect on engagement at +30 and +180 days: (1) staggered-adoption difference-in-differences/event-study; (2) propensity-score matched IPW/DR or a synthetic control at the user/region level. For each, write the estimand, key assumptions, diagnostics (pre-trends, covariate balance, negative controls), and how you will handle interference (household spillovers) and seasonality. C) Leverage exogenous variation from regional stockouts/shipping delays as an instrument: argue relevance and exclusion, and outline falsification tests. D) Provide a back-of-the-envelope power/MDE given 6% adoption and historical variance: what sample size or horizon is needed to detect a 3% lift?

Loading comments...