Estimate ATE of personalization on streaming
Company: Netflix
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
You are given a user-level dataset from an online experiment that randomized **personalization** (treatment) vs **no personalization** (control).
Assume one row per user with the following columns:
- `user_id` (string/int)
- `treat` (0/1): randomized assignment to personalization
- `minutes_streamed` (float): total minutes streamed during the 7-day post-assignment window
- Optional pre-treatment covariates (may include irrelevant/noisy variables): e.g., `country`, `device_type`, `tenure_days`, `prior_7d_minutes`, `is_premium`, etc.
Task:
1) Estimate the **Average Treatment Effect (ATE)** of personalization on `minutes_streamed`.
2) Report a 95% confidence interval and describe at least one valid way to compute it.
3) Explain (briefly) whether and how you would use the provided covariates (including why adding irrelevant covariates can still be OK / not OK).
Assumptions:
- Randomization is at the user level; no interference (SUTVA).
- Use a two-sided 95% CI.
- If you use regression, treat `treat` as the only post-treatment variable; all other covariates are pre-treatment.
Quick Answer: This question evaluates causal inference and experimental-analysis competencies by asking for estimation of the average treatment effect (ATE) of personalization on minutes streamed, reporting a 95% confidence interval, and reasoning about the use of pre-treatment covariates, and it sits in the Analytics & Experimentation domain for a Data Scientist role. It is commonly asked to assess application of randomized experiment analysis and statistical inference while probing conceptual understanding of causal assumptions and covariate adjustment, testing both conceptual understanding and practical application.