Diagnose a watch-time drop and design experiments

A short‑video app rolled out a new preloading strategy to 30% of traffic on 2025-08-20. Among new users (accounts created in the last 7 days), product analytics shows a −6% change in average daily watch time, while crash rate decreased by 0.2 pp and average initial video start latency improved by 80 ms. Design an end‑to‑end approach: (a) Define primary, secondary, and guardrail metrics. Justify each and propose segmentations (e.g., device, network, country, cohort age, entry surface). Specify exact formulas and units. (b) Outline an A/B test plan: unit of randomization, bucketing, exposure rules, test length, and how you would handle heavy‑tailed watch time (e.g., winsorization, log‑transform, robust estimators). (c) Estimate per‑variant sample size for detecting a +3% lift in mean daily watch time with α=0.05 (two‑sided) and 80% power, assuming baseline mean = 14 min, SD = 18 min, independent users, and equal allocation. Show the formula you’d use and what additional assumptions you need if using nonparametric tests. (d) Specify guardrails (e.g., crash rate, time‑to‑first‑frame, data usage) and stopping rules. How would you handle novelty effects, weekday/seasonality, and experiment mis‑randomization checks (e.g., A/A, covariate balance tests)? (e) If the feature was partially rolled out by region before the test, propose a difference‑in‑differences or CUPED/regression‑adjusted analysis. State key identifying assumptions and how you’d validate them.