Design a weapon-ad harmful content detection system

Scenario

You are building a system for an ads platform that must detect advertisements that contain weapons (e.g., guns, knives) and prevent policy-violating ads from being served.

Ads may include:

• Images (single/multi-image)
• Video
• Text (title/description)
• Landing page URL/content

What to design

Design an end-to-end harmful content detection system that:

1. Screens ads at submission time (and/or pre-serve) with low latency.
2. Supports high throughput (large ad volume).
3. Provides a decision: approve / reject / send to human review .
4. Handles ambiguous cases, adversarial attempts, and policy changes.
5. Enables continuous improvement via feedback and retraining.

Deliverables

In your design, cover:

• Functional + non-functional requirements (latency, throughput, availability)
• High-level architecture and key components
• Model approach (multi-modal), feature extraction, inference serving
• Data labeling, human review workflow, and feedback loops
• Monitoring/metrics and quality evaluation (precision/recall, false positives)
• Safety, privacy, and abuse considerations