Design a harmful content detection system
Overview
This question evaluates proficiency in ML system design for harmful-content detection, assessing competencies in multimodal modeling, taxonomy and labeling strategies, data strategy, privacy-aware architecture, real-time inference, human-in-the-loop decisioning, and operational reliability for global consumer platforms.
System Design: End-to-End Harmful Content Detection (Multilingual, Multimodal)
Context
You are designing a safety system for a large, mobile-first, ephemeral, consumer social platform with a global user base including teens. Users share multimodal content across surfaces such as 1:1/group chat, short-form video, stories, lenses/AR effects, and live audio/video. The platform must detect and act on harmful content in near real-time while respecting privacy, regional policies, and user trust.
Requirements
Design an end-to-end harmful content detection system that covers the following:
- Taxonomy and Labeling
- Define a practical taxonomy (e.g., hate, self-harm, sexual content, violence) with subcategories and severity tiers.
- Provide labeling guidelines and quality controls for annotation.
- Define multilingual and multimodal scope (text, image, audio, video) and context rules (e.g., multi-turn chat, text-in-image).
- Modeling and Data Strategy
- Propose model choices: keyword baselines, classical ML, modern transformers, and multimodal encoders.
- Training data strategy: data collection, active learning, long-tail sampling, debiasing, and handling sensitive classes.
- Inference and Decisioning
- Inference architecture: streaming vs batch, on-device vs server, latency targets by surface.
- Thresholding, severity tiers, and policy actions (block, age-gate, interstitial, downrank, quarantine).
- Human-in-the-loop: triage queues, escalation, appeals/override flows.
- Explainability requirements for moderators and user-facing transparency.
- Risk, Privacy, and Reliability
- Adversarial behavior: evasion (obfuscation, text-in-image, coded language), prompt injection (for any LLM components), model hardening.
- Privacy and safety constraints: data minimization, retention, encryption, age-appropriate design.
- Fairness and error costs: precision/recall trade-offs by class and region; group fairness.
- Monitoring, drift detection, A/B rollout, and feedback loops for continuous improvement.
Deliver a cohesive design that integrates these components into an operational system, explaining assumptions and trade-offs.
Solution
Login required