Choose Randomization Unit and Mitigate Network Effects
A/B Test Design for a New Messenger Feature with Network Effects
Context
You are designing an A/B test for a new Messenger feature in a social messaging app. Users communicate via one-to-one and group conversations. Because people interact, there is a risk that a participant's assignment could influence their peers' outcomes (network interference), violating the standard "no interference" assumption of A/B tests.
Tasks
-
Choose an appropriate randomization unit (e.g., user-level, conversation/thread-level, cluster/graph-level) and justify your choice given potential network effects.
-
Describe practical diagnostics and statistical tests you would use to detect network effects if they occur.
-
Propose design and analysis methods to mitigate bias from network effects and to estimate both direct and spillover effects. Where useful, mention methods such as CUPED and difference-in-differences.
Constraints & Assumptions
-
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
-
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
-
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask
-
Clarify the random variables, distributional assumptions, independence assumptions, and desired output.
-
Show enough derivation for the interviewer to follow the reasoning.
-
Explain how you would validate the result with simulation or sensitivity checks.
What a Strong Answer Covers
-
A correct setup with definitions, formulas, and boundary conditions.
-
A step-by-step derivation or estimation plan.
-
Interpretation of the result, including uncertainty and practical limitations.
-
Checks for assumptions, edge cases, and numerical stability.
Follow-up Questions
-
How would the result change if the assumptions were relaxed?
-
Can you verify the answer with a simulation?
-
What is the most likely source of estimation error?