Evaluate Auto-Play Impact with Key Metrics and Experiment Design
Overview
This question evaluates a data scientist's competency in experiment design, metric selection, causal inference, segmentation, and trade-off analysis for product changes such as auto-play, including defining north-star and guardrail metrics.
Uber
Jul 12, 2025, 6:59 PM
Data Scientist
Technical Screen
Analytics & Experimentation
22
0
Streaming Auto-Play Experiment: Metrics and Design
Context
A streaming platform is considering auto-playing the next episode to increase user engagement. The company wants to evaluate if auto-play meaningfully improves engagement without harming user satisfaction or retention.
Task
- Define the primary (north-star) metric(s) to evaluate auto-play and the key guardrail metrics to protect user experience and business health.
-
Design an experiment to measure impact:
- State the hypothesis and success criteria.
- Specify the unit of randomization and any stratification.
- Outline duration, sample size approach, and analysis plan.
- If results show churn increases among power users while total watch time rises overall, explain how you would decide whether to launch.
Hints
- Discuss trade-offs, segmentation, and how the north-star metric guides decisions.
- Consider user-level vs. account-level randomization, exposure/triggered analysis, and guardrails for retention and satisfaction.
Solution
Login required
Show