Evaluate Impact of New Roblox Homepage Tab
Overview
This question evaluates a data scientist's skills in experimental design, causal inference, metric selection (primary outcomes and guardrails), power analysis, segmentation, and confounder mitigation within product analytics.
Roblox
Jul 12, 2025, 6:59 PM
Data Scientist
Technical Screen
Analytics & Experimentation
45
0
Measuring the Impact of Replacing a Homepage Tab
Scenario
Roblox plans to replace an existing homepage tab with a new one across eligible users. The goal is to understand whether the new tab improves user engagement and downstream outcomes without harming core health metrics.
Task
Design an evaluation plan that includes:
- Hypotheses
- Primary outcome metrics and guardrail (safety) metrics
- Experiment design (unit of randomization, allocation, instrumentation)
- Sample size, duration, and ramp plan
- Segmentation
- Success criteria and decision rules
- Confounders and how you will mitigate them
Hints: Consider the engagement funnel (CTR → launches → playtime → retention), power analysis, guardrails, and potential confounders.
Solution
Login required
Show