Test 15s to 60s video length change

Quick Overview

This question evaluates a data scientist's competency in experimental design, causal inference, metric definition and instrumentation, power/MDE calculations, interference and contamination handling on multi‑sided platforms, and rollout planning.

Test 15s to 60s video length change

Company: TikTok

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

Product proposes changing default video length from 15s to 60s. Design the experiment. Specify: (1) unit of randomization and exposure rules to avoid contamination (creators, viewers, or geo; how to handle multi-sided interactions); (2) primary outcomes and guardrails (e.g., completion rate, total watch time per user, creator retention, session length, ad revenue per mille, latency/crash rate); (3) traffic allocation, ramp schedule, and expected MDE with power assumptions; (4) how you will detect novelty and learning effects and set experiment duration; (5) checks for SUTVA/interference and cluster variance; and (6) a rollout plan if results pass. Be concrete about metric definitions and instrumentation.

Quick Answer: This question evaluates a data scientist's competency in experimental design, causal inference, metric definition and instrumentation, power/MDE calculations, interference and contamination handling on multi‑sided platforms, and rollout planning.

|Home/Analytics & Experimentation/TikTok
TikTok logo
TikTok
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteAnalytics & Experimentation
7
0

Experiment Design: Change Default Creator Video Length from 15s to 60s

Context

You are designing an A/B test for a short‑form video platform with two sides: creators (supply) and viewers (demand). Product proposes changing the default preselected recording length in the creation UI from 15 seconds to 60 seconds. The hypothesis is that this nudge will increase the supply and consumption of longer videos, affecting viewer engagement, monetization, and system health.

Task

Design the experiment and cover the following:

  1. Unit of randomization and exposure rules to avoid contamination (creators, viewers, or geo; how to handle multi‑sided interactions).
  2. Primary outcomes and guardrails (e.g., completion rate, total watch time per user, creator retention, session length, ad revenue per mille, latency/crash rate), with concrete metric definitions and instrumentation.
  3. Traffic allocation, ramp schedule, and expected MDE with power assumptions.
  4. How to detect novelty and learning effects and set experiment duration.
  5. Checks for SUTVA/interference and cluster variance.
  6. A rollout plan if results pass.
Loading comments...