Experiment Design: Measuring the Impact of a Bot‑Mitigation System

Context

You are evaluating a production change to a large social platform that hides or rate‑limits comments from suspected bot accounts. The goal is to improve the human experience without harming creators or platform health. You must design an experiment (and fallback causal strategy) that addresses interference, metrics, eligibility, power, novelty/adaptation, and diagnostics.

Tasks

1. Randomization unit and interference

• Choose and justify the randomization unit (account‑level vs. graph/community clusters) to limit spillovers.
• Describe how you'll handle interference via the social graph (viewers, commenters, creators).
• Propose a stepped‑wedge ramp plan.

2. Primary success metrics and guardrails

• Primary: human‑visible comments per human DAU, human creator retention, report rates.
• Define guardrails: new‑user activation, moderation backlog, latency, etc.

3. Eligibility and exclusions

• Define which users/geos are eligible.
• Specify how to handle already‑flagged accounts and high‑risk geos.
• Include a pre‑exposure period and use CUPED to reduce variance.

4. Power and duration

• Baseline: human‑visible comments/DAU = 5.5.
• MDEs: −2% total comments; +1% human‑visible comments.
• Outline sample‑size and duration calculations for a 14‑day test with day‑level clustering.

5. Novelty and adaptation

• Plan for novelty decay, contamination from bot migration, and adversarial learning.

6. Fallback when A/B is infeasible

• Use difference‑in‑differences with synthetic controls and high‑frequency pre‑trends checks.

7. Diagnostics if false positives rise

• Define on‑experiment holdouts and stratified analysis (new vs. veteran users, creators vs. consumers).

Deliver an analysis plan with stop/go criteria.