Measure impact of bot mitigation via experiment

You will measure the impact of deploying a bot-mitigation system that hides or rate-limits suspected bots’ comments. Design an experiment covering: 1) Randomization unit and interference: justify account-level vs cluster (graph/community) randomization to limit spillovers; propose a stepped-wedge ramp. 2) Primary success metrics: human-visible comments per human DAU, human creator retention, report rates; define guardrails (new-user activation, moderation backlog, latency). 3) Eligibility and exclusions: how to handle already-flagged accounts and high-risk geos; pre-exposure period and CUPED to reduce variance. 4) Power: assuming baseline human-visible comments/DAU = 5.5 and MDE = −2% total comments but +1% human-visible comments, outline sample-size and duration calculations for a 14-day test with day-level clustering. 5) Novelty and adaptation: plan for novelty decay, contamination from bot migration, and adversarial learning. 6) Fallback when A/B is infeasible: difference-in-differences with synthetic controls and high-frequency pre-trends checks. 7) Diagnostics if FP rises: define on-experiment holdouts and stratified analysis (new vs veteran users, creators vs consumers). Deliver an analysis plan and stop/go criteria.