Test 15s to 60s video length change
Overview
This question evaluates a data scientist's competency in experimental design, causal inference, metric definition and instrumentation, power/MDE calculations, interference and contamination handling on multi‑sided platforms, and rollout planning.
TikTok
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Experiment Design: Change Default Creator Video Length from 15s to 60s
Context
You are designing an A/B test for a short‑form video platform with two sides: creators (supply) and viewers (demand). Product proposes changing the default preselected recording length in the creation UI from 15 seconds to 60 seconds. The hypothesis is that this nudge will increase the supply and consumption of longer videos, affecting viewer engagement, monetization, and system health.
Task
Design the experiment and cover the following:
- Unit of randomization and exposure rules to avoid contamination (creators, viewers, or geo; how to handle multi‑sided interactions).
- Primary outcomes and guardrails (e.g., completion rate, total watch time per user, creator retention, session length, ad revenue per mille, latency/crash rate), with concrete metric definitions and instrumentation.
- Traffic allocation, ramp schedule, and expected MDE with power assumptions.
- How to detect novelty and learning effects and set experiment duration.
- Checks for SUTVA/interference and cluster variance.
- A rollout plan if results pass.
Solution
Login required
Show