Design a creator posting-frequency experiment
Overview
This question evaluates experimental design and causal inference competencies for a Data Scientist, including precise metric definition, randomization and interference reasoning, eligibility and ITT versus triggered analyses, sample‑size and power estimation, skewed count modeling, sequential monitoring and heterogeneity analysis within the Analytics & Experimentation domain. It is commonly asked because interviewers use it to assess end‑to‑end experimental thinking that balances statistical rigor, operational constraints and guardrails, and it requires both practical application (power calculations, triggers, model choices) and conceptual understanding (interference, SRM root‑cause reasoning); summary provided in English.
You’re on the Creator Growth (PGC) team of a short‑video platform. Product proposes a push/email nudge expected to raise creators’ weekly posting frequency by 10%.
Design an experiment and analysis plan:
- Precisely define: (a) Primary metric = creator‑week posts per active creator; (b) Secondary = creator retention, viewer engagement; (c) Guardrails = viewer dissatisfaction/complaint rate, abuse reports, latency/crash rate. Write exact formulas and units for each.
- Choose the randomization unit and targeting (creator-level vs geo/graph clusters). Justify in terms of interference/spillover (e.g., shared viewers, duet/remix features) and operational complexity.
- Eligibility/triggering: define which creators are eligible (e.g., ≥1 post in the prior 28 days), when they are considered “treated”, and how you’ll handle creators who never open the nudge. Contrast ITT vs triggered analysis and what you ship on.
- Power/duration: With baseline mean 1.8 posts/week (sd 2.5) among eligible creators, two‑sided α=0.05, power=90%, MDE=+4% relative on the primary metric, equal allocation—estimate required sample size and test length. State assumptions and show formulas or approximations you use.
- Analysis: specify pre‑period adjustment (e.g., CUPED), model choice for skew/zeros (e.g., log(1+x) vs Negative Binomial), heterogeneity by geography and creator tenure, and your SRM checks. It’s 2025‑09‑01: if SRM triggers, list the top three root‑cause checks you’d run immediately.
- Novelty/fatigue: propose a ramp strategy and a sequential‑monitoring plan that controls Type I error.
- Suppose results show US +3% lift, BR −2% lift, and global +1% lift. What do you ship, where, and what follow‑ups do you run to validate the geo divergence?