You’re on the Creator Growth (PGC) team of a short‑video platform. Product proposes a push/email nudge expected to raise creators’ weekly posting frequency by 10%.
Design an experiment and analysis plan:
-
Precisely define: (a) Primary metric = creator‑week posts per active creator; (b) Secondary = creator retention, viewer engagement; (c) Guardrails = viewer dissatisfaction/complaint rate, abuse reports, latency/crash rate. Write exact formulas and units for each.
-
Choose the randomization unit and targeting (creator-level vs geo/graph clusters). Justify in terms of interference/spillover (e.g., shared viewers, duet/remix features) and operational complexity.
-
Eligibility/triggering: define which creators are eligible (e.g., ≥1 post in the prior 28 days), when they are considered “treated”, and how you’ll handle creators who never open the nudge. Contrast ITT vs triggered analysis and what you ship on.
-
Power/duration: With baseline mean 1.8 posts/week (sd 2.5) among eligible creators, two‑sided α=0.05, power=90%, MDE=+4% relative on the primary metric, equal allocation—estimate required sample size and test length. State assumptions and show formulas or approximations you use.
-
Analysis: specify pre‑period adjustment (e.g., CUPED), model choice for skew/zeros (e.g., log(1+x) vs Negative Binomial), heterogeneity by geography and creator tenure, and your SRM checks. It’s 2025‑09‑01: if SRM triggers, list the top three root‑cause checks you’d run immediately.
-
Novelty/fatigue: propose a ramp strategy and a sequential‑monitoring plan that controls Type I error.
-
Suppose results show US +3% lift, BR −2% lift, and global +1% lift. What do you ship, where, and what follow‑ups do you run to validate the geo divergence?