Design weapon-selling ad detection from posts
Overview
This question evaluates a candidate's ability to design a production-scale multimodal ML system for detecting weapon-selling ads, testing competencies in policy definition, data and labeling strategy, multimodal feature/model design, robustness to evasion, latency and precision/recall trade-offs, human-in-the-loop workflows, monitoring, and privacy-aware deployment. It is commonly asked in the ML system design domain to assess architectural thinking for safety-critical content moderation and operational MLOps, requiring both conceptual understanding of policy and trade-offs and practical application for deployment, evaluation, and iteration.
ML System Design: Detect weapon-selling ads from user posts
You work on a platform with user-generated content (UGC): posts may include text, images, video thumbnails, user metadata, and outbound links.
Goal: Detect and take action on posts that are attempting to sell weapons (e.g., firearms, ammunition, certain knives depending on policy). The system should distinguish:
- benign mentions (news, education, gaming)
- weapon possession or display (not necessarily selling)
- selling intent / transaction facilitation (price, contact info, shipping, “DM to buy”, marketplaces)
Requirements
- Near-real-time moderation support (e.g., P95 latency target you define).
- High precision at the policy-action threshold (avoid wrongful takedowns), while maintaining strong recall for harmful content.
- Robust to evasion (misspellings, code words, images with text overlays, obfuscated contact info).
- Human review workflow for uncertain cases.
- Logging/monitoring and a plan to iterate with feedback.
Deliverables
Describe:
- Problem clarification and policy definition (what counts as “weapon” and what counts as “selling”).
- Data and labeling strategy.
- Feature/model approach (text + image + metadata; selling intent signals).
- Training, evaluation metrics, and thresholding.
- Deployment architecture (online inference, human-in-the-loop, retraining).
- Monitoring, abuse handling, and privacy considerations.
Solution
Login required