PracHub
QuestionsPremiumLearningGuidesCheatsheetNEWCoaches
|Home/ML System Design/Snapchat

Design a harmful content detection system

Last updated: Mar 29, 2026

Quick Overview

This question evaluates proficiency in ML system design for harmful-content detection, assessing competencies in multimodal modeling, taxonomy and labeling strategies, data strategy, privacy-aware architecture, real-time inference, human-in-the-loop decisioning, and operational reliability for global consumer platforms.

  • hard
  • Snapchat
  • ML System Design
  • Machine Learning Engineer

Design a harmful content detection system

Company: Snapchat

Role: Machine Learning Engineer

Category: ML System Design

Difficulty: hard

Interview Round: Onsite

Design an end-to-end harmful content detection system. Define the taxonomy (e.g., hate, self-harm, sexual content, violence), labeling guidelines and quality controls, and multilingual/multimodal scope (text, image, audio, video). Propose model choices (keyword baselines, classical ML, transformers, multimodal encoders) and training data strategy (collection, active learning, long-tail sampling, debiasing). Specify inference architecture (streaming vs. batch), thresholds and severity tiers, human-in-the-loop review, appeals/override flows, and explainability requirements. Address adversarial behavior (evasion, prompt injection), privacy and safety constraints, fairness and error costs (precision/recall trade-offs by class and region), monitoring and drift detection, A/B rollout, and feedback loops for continuous improvement.

Quick Answer: This question evaluates proficiency in ML system design for harmful-content detection, assessing competencies in multimodal modeling, taxonomy and labeling strategies, data strategy, privacy-aware architecture, real-time inference, human-in-the-loop decisioning, and operational reliability for global consumer platforms.

Related Interview Questions

  • Design User Embedding Semantic Search - Snapchat (medium)
  • Design a Family-Friendly Listing Classifier - Snapchat (medium)
  • Design a video recommendation system - Snapchat (medium)
  • Design an ads ranking ML system - Snapchat (medium)
  • Design short-video retrieval with sparse text - Snapchat (medium)
Snapchat logo
Snapchat
Sep 6, 2025, 12:00 AM
Machine Learning Engineer
Onsite
ML System Design
6
0

System Design: End-to-End Harmful Content Detection (Multilingual, Multimodal)

Context

You are designing a safety system for a large, mobile-first, ephemeral, consumer social platform with a global user base including teens. Users share multimodal content across surfaces such as 1:1/group chat, short-form video, stories, lenses/AR effects, and live audio/video. The platform must detect and act on harmful content in near real-time while respecting privacy, regional policies, and user trust.

Requirements

Design an end-to-end harmful content detection system that covers the following:

  1. Taxonomy and Labeling
  • Define a practical taxonomy (e.g., hate, self-harm, sexual content, violence) with subcategories and severity tiers.
  • Provide labeling guidelines and quality controls for annotation.
  • Define multilingual and multimodal scope (text, image, audio, video) and context rules (e.g., multi-turn chat, text-in-image).
  1. Modeling and Data Strategy
  • Propose model choices: keyword baselines, classical ML, modern transformers, and multimodal encoders.
  • Training data strategy: data collection, active learning, long-tail sampling, debiasing, and handling sensitive classes.
  1. Inference and Decisioning
  • Inference architecture: streaming vs batch, on-device vs server, latency targets by surface.
  • Thresholding, severity tiers, and policy actions (block, age-gate, interstitial, downrank, quarantine).
  • Human-in-the-loop: triage queues, escalation, appeals/override flows.
  • Explainability requirements for moderators and user-facing transparency.
  1. Risk, Privacy, and Reliability
  • Adversarial behavior: evasion (obfuscation, text-in-image, coded language), prompt injection (for any LLM components), model hardening.
  • Privacy and safety constraints: data minimization, retention, encryption, age-appropriate design.
  • Fairness and error costs: precision/recall trade-offs by class and region; group fairness.
  • Monitoring, drift detection, A/B rollout, and feedback loops for continuous improvement.

Deliver a cohesive design that integrates these components into an operational system, explaining assumptions and trade-offs.

Solution

Show

Comments (0)

Sign in to leave a comment

Loading comments...

Browse More Questions

More ML System Design•More Snapchat•More Machine Learning Engineer•Snapchat Machine Learning Engineer•Snapchat ML System Design•Machine Learning Engineer ML System Design
PracHub

Master your tech interviews with 7,500+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.