How to estimate a feature’s causal impact on time spent
Company: Roblox
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
You work on a Roblox-like game platform. A new product change ("feature") is rolled out and you want to estimate its **causal impact** on user engagement, measured as **daily time spent (minutes per user-day)**.
However, the rollout is **not fully randomized**:
- The feature was enabled first for some platforms/regions and later for others.
- More engaged users may be more likely to receive/enable the feature earlier.
- Time spent has strong seasonality (day-of-week) and a general upward/downward trend.
You have event-level logs aggregated to a user-day table:
- `user_id`
- `date` (in UTC)
- `minutes_spent`
- `feature_enabled` (1 if the feature is enabled for that user on that date)
- User attributes (e.g., `country`, `platform`, `account_age_days`)
Task:
1) Describe how you would determine whether the feature affects `minutes_spent`, including how you would handle **confounding**.
2) If you choose **Difference-in-Differences (DiD)**, specify:
- What are the treatment/control groups and pre/post windows?
- The key identifying assumption(s) and how you would check them.
- A regression specification you would run and what coefficient answers the question.
3) List common failure modes (e.g., violated assumptions, interference) and at least **two alternative approaches** if DiD is not credible.
State any additional assumptions you need and what outputs (tables/plots) you would show to stakeholders.
Quick Answer: Evaluates causal inference and time-series experimentation skills—specifically handling confounding, staggered rollouts, seasonality, and credible treatment-effect identification—in the Analytics & Experimentation domain at a mid-to-senior data scientist abstraction level.