Experiment Design: Stricter Spam Filter Impact on Friend Requests

Context

You run a social app with a friend-request system. A stricter spam filter will score and potentially block outgoing requests before delivery to recipients. You want to measure its impact on same-day acceptance behavior while protecting sender and recipient experience.

Assumptions for clarity (adjust if needed):

• A "request" is created at send_time and is either delivered (passes filter) or blocked (filtered as spam).
• A delivered request can be accepted at any later time.
• Dates and windows use UTC boundaries.

Tasks

a) Hypotheses

• Define a primary hypothesis on the same-day acceptance rate.
• Define at least two guardrail hypotheses (e.g., total requests sent/delivered, approval latency, false-positive spam rate).
• State the null and alternative precisely.

b) Experimental unit and randomization scheme

• Propose the experimental unit and a randomization approach that mitigates network interference (cluster by requester vs. recipient, etc.).
• Justify the choice and discuss spillover risks.

c) Metrics and windows

• Define primary and secondary metrics with exact measurement windows and UTC date boundaries.
• Explain how to handle approvals that occur on days after the request is sent.

d) Sample size and power plan

• Provide assumed baseline same-day acceptance, MDE, variance source, test duration.
• Describe sequential looks and Type I error control.

e) Incomplete spam labels

• Explain how incomplete/unknown ground-truth labels could bias metrics.
• Propose two mitigation strategies (e.g., unknown bucket + sensitivity bounds; propensity/inverse-probability weighting if MAR) and how you would report adjusted results.