Design an ML System to Detect Weapon-Related Content in Social Media Posts
Company: Meta
Role: Machine Learning Engineer
Category: ML System Design
Difficulty: medium
Interview Round: Onsite
Design an ML system that detects weapon-related content in posts on a large social media platform, so that violating posts are removed, demoted or sent to human review. The report names only the topic, and the discussion was interleaved with questions on the ML fundamentals behind the design, so expect to justify each modeling choice. The policy scope, the content types and the scale are open: settle them with the interviewer before designing.
```hint Object or intent
A detector that finds a gun in the pixels is only part of the job. Think about which posts with a visible weapon are fine, and which posts without one still break the policy.
```
```hint Rare positives
Think about how small the share of violating posts is, and what that does to the training set, the loss and the metrics you report.
```
### Clarifying Questions
- What exactly violates the policy: any visible weapon, or only selling, trading, threatening or promoting violence? Are hunting, sport, news, toy and video-game weapons allowed?
- Which content types are in scope: images, videos, captions, comments, live streams?
- What happens on a detection: automatic removal, reduced distribution or human review, and how many posts can reviewers handle a day?
- How many posts are checked per day, and must the check finish before a post is published?
- Which error costs more: removing a legitimate post or missing a violating one?
### What a Strong Answer Covers
- Policy-driven framing: label classes, content types, and actions tied to confidence levels
- Training data from reviewer decisions and random samples, hard negatives, label noise and policy changes
- Image, video, text and account models with fusion and handling of missing modalities
- The ML fundamentals behind the choices: class imbalance, loss, calibration, metrics for rare positives, overfitting
- A cost-aware serving cascade, with thresholds for removal and review that respect reviewer capacity
- Offline and online evaluation, adversarial evasion, monitoring and retraining
### Follow-up Questions
- Violating posts are a tiny fraction of all posts. Why can accuracy and even ROC AUC look excellent for a model that is useless in practice, and what would you report instead?
- How would you set separate thresholds for automatic removal and for human review, and keep them valid after the model is retrained?
- Sellers start blurring, cropping or covering weapons with stickers and switching to coded words. How do you notice the shift and adapt?
- Some posts have only an image, some only text. How does your fusion model handle a missing modality?
Overview: An ML system design question on detecting weapon-related content in images, videos and text posted to a large social platform. It tests policy framing, training data for rare positives, multimodal models, calibrated thresholds for removal and human review, adversarial evasion and the ML fundamentals behind each choice.