You will measure the impact of deploying a bot-mitigation system that hides or rate-limits suspected bots’ comments. Design an experiment covering: 1) Randomization unit and interference: justify account-level vs cluster (graph/community) randomization to limit spillovers; propose a stepped-wedge ramp. 2) Primary success metrics: human-visible comments per human DAU, human creator retention, report rates; define guardrails (new-user activation, moderation backlog, latency). 3) Eligibility and exclusions: how to handle already-flagged accounts and high-risk geos; pre-exposure period and CUPED to reduce variance. 4) Power: assuming baseline human-visible comments/DAU = 5.5 and MDE = −2% total comments but +1% human-visible comments, outline sample-size and duration calculations for a 14-day test with day-level clustering. 5) Novelty and adaptation: plan for novelty decay, contamination from bot migration, and adversarial learning. 6) Fallback when A/B is infeasible: difference-in-differences with synthetic controls and high-frequency pre-trends checks. 7) Diagnostics if FP rises: define on-experiment holdouts and stratified analysis (new vs veteran users, creators vs consumers). Deliver an analysis plan and stop/go criteria.

This question evaluates experiment design and causal inference skills for platform analytics, covering interference management, metric and guardrail definition, power and duration calculations, eligibility and exclusions, novelty/adaptation considerations, and diagnostics for bot‑mitigation interventions.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Measure impact of bot mitigation via experiment

Experiment Design: Measuring the Impact of a Bot‑Mitigation System

Context

You are evaluating a production change to a large social platform that hides or rate‑limits comments from suspected bot accounts. The goal is to improve the human experience without harming creators or platform health. You must design an experiment (and fallback causal strategy) that addresses interference, metrics, eligibility, power, novelty/adaptation, and diagnostics.

Tasks

Randomization unit and interference

Choose and justify the randomization unit (account‑level vs. graph/community clusters) to limit spillovers.
Describe how you'll handle interference via the social graph (viewers, commenters, creators).
Propose a stepped‑wedge ramp plan.

Primary success metrics and guardrails

Primary: human‑visible comments per human DAU, human creator retention, report rates.
Define guardrails: new‑user activation, moderation backlog, latency, etc.

Eligibility and exclusions

Define which users/geos are eligible.
Specify how to handle already‑flagged accounts and high‑risk geos.
Include a pre‑exposure period and use CUPED to reduce variance.

Power and duration

Baseline: human‑visible comments/DAU = 5.5.
MDEs: −2% total comments; +1% human‑visible comments.
Outline sample‑size and duration calculations for a 14‑day test with day‑level clustering.

Novelty and adaptation

Plan for novelty decay, contamination from bot migration, and adversarial learning.

Fallback when A/B is infeasible

Use difference‑in‑differences with synthetic controls and high‑frequency pre‑trends checks.

Diagnostics if false positives rise

Define on‑experiment holdouts and stratified analysis (new vs. veteran users, creators vs. consumers).

Deliver an analysis plan with stop/go criteria.

Experiment Design: Measuring the Impact of a Bot‑Mitigation System

Context

Tasks

Randomization unit and interference

Choose and justify the randomization unit (account‑level vs. graph/community clusters) to limit spillovers.
Describe how you'll handle interference via the social graph (viewers, commenters, creators).
Propose a stepped‑wedge ramp plan.

Primary success metrics and guardrails

Primary: human‑visible comments per human DAU, human creator retention, report rates.
Define guardrails: new‑user activation, moderation backlog, latency, etc.

Eligibility and exclusions

Define which users/geos are eligible.
Specify how to handle already‑flagged accounts and high‑risk geos.
Include a pre‑exposure period and use CUPED to reduce variance.

Power and duration

Baseline: human‑visible comments/DAU = 5.5.
MDEs: −2% total comments; +1% human‑visible comments.
Outline sample‑size and duration calculations for a 14‑day test with day‑level clustering.

Novelty and adaptation

Plan for novelty decay, contamination from bot migration, and adversarial learning.

Fallback when A/B is infeasible

Use difference‑in‑differences with synthetic controls and high‑frequency pre‑trends checks.

Diagnostics if false positives rise

Define on‑experiment holdouts and stratified analysis (new vs. veteran users, creators vs. consumers).

Deliver an analysis plan with stop/go criteria.

Measure impact of bot mitigation via experiment

Quick Overview

Experiment Design: Measuring the Impact of a Bot‑Mitigation System

Context

Tasks

Solution

Submit Your Answer

Measure impact of bot mitigation via experiment

Quick Overview

Experiment Design: Measuring the Impact of a Bot‑Mitigation System

Context

Tasks

Solution

Submit Your Answer