Design an unsafe content detection system

Scenario

You are building a system that detects and mitigates unsafe user-generated content (UGC) on a large platform.

Unsafe content can include: hate/harassment, sexual content, self-harm, violence, spam/scams, and policy-violating content.

Task

Design an end-to-end ML system to:

1. Detect unsafe content at upload/post time and after posting (e.g., via reports or virality).
2. Take actions (allow, down-rank, blur/interstitial, age-gate, block, queue for human review).
3. Support multiple modalities as applicable (text, images, video, audio), and account for multilingual content.

Requirements (state assumptions if needed)

• Latency: real-time decision for the user-facing publish path.
• Scale: high QPS and large content volume.
• Quality: minimize both false negatives (missed unsafe content) and false positives (incorrect takedowns).
• Reliability & safety: auditing, appeal workflow, and policy evolution.

Interviewer prompts to expect

• What are the modeling approaches and feature signals?
• What is your serving architecture (services, caches, async vs sync)?
• How do you evaluate (offline metrics + online guardrails)?
• How do you handle concept drift/adversaries , human-in-the-loop, and retraining?