Meta Ad Reporting Success Metrics: Report Precision, Missed Violations, Wrongful Removals

You are a product manager at Meta. People can report an ad they see in Meta's apps. How would you measure whether this ad-reporting feature is successful, taking both false positives and false negatives into account? ```hint Follow one report Trace a report from the tap to the final decision on the ad, and decide what counts as a positive at each step before you pick metrics. ``` ### Clarifying Questions - Is the feature mainly for flagging ads that break advertising rules (scams, misleading claims, offensive content), or also for ads that are simply irrelevant or repetitive? - What should count as a false positive: a report on an ad that breaks no rule, or an ad wrongly removed after review? And a false negative: a violating ad that nobody reports, or a valid report that review dismisses? - Should I cover every app and placement where Meta shows ads, or one surface such as Feed? - Do reports trigger automated action directly, or does every action go through review? ### What a Strong Answer Covers - The feature's goal and the path from report to review to action on the ad - False positives and false negatives defined at each stage, with a way to measure each, including violating ads nobody reports - A primary metric with supporting metrics for usage, report quality and speed - Guardrails for advertisers and revenue, and for abuse such as coordinated false reporting - How the cost of each error depends on the severity of the violation ### Follow-up Questions - Reports per thousand ad impressions doubled after the reporting menu was redesigned. How do you tell whether ads got worse or reporting got easier? - A wave of coordinated reports hits one advertiser's ads. How should your metrics and the system detect and handle it? - How would you set a different tolerance for missed violations on scam ads than on ads people only find irrelevant?