Design an unsafe content detection system
Company: Pinterest
Role: Machine Learning Engineer
Category: ML System Design
Difficulty: hard
Interview Round: Onsite
## Scenario
You are building a system that detects and mitigates unsafe user-generated content (UGC) on a large platform.
Unsafe content can include: hate/harassment, sexual content, self-harm, violence, spam/scams, and policy-violating content.
## Task
Design an end-to-end ML system to:
1. **Detect** unsafe content at upload/post time and after posting (e.g., via reports or virality).
2. **Take actions** (allow, down-rank, blur/interstitial, age-gate, block, queue for human review).
3. Support **multiple modalities** as applicable (text, images, video, audio), and account for multilingual content.
## Requirements (state assumptions if needed)
- **Latency:** real-time decision for the user-facing publish path.
- **Scale:** high QPS and large content volume.
- **Quality:** minimize both false negatives (missed unsafe content) and false positives (incorrect takedowns).
- **Reliability & safety:** auditing, appeal workflow, and policy evolution.
## Interviewer prompts to expect
- What are the **modeling approaches** and feature signals?
- What is your **serving architecture** (services, caches, async vs sync)?
- How do you **evaluate** (offline metrics + online guardrails)?
- How do you handle **concept drift/adversaries**, human-in-the-loop, and retraining?
Quick Answer: This question evaluates a candidate's competency in end-to-end machine learning system design for unsafe user-generated content detection, covering multimodal modeling, real-time inference and serving, scalability, reliability, evaluation metrics, policy-driven actions, and human-in-the-loop workflows.