Design a Multimodal Classifier for Unsafe Pins Using Images, Text and User Reports

Quick Overview

Design a machine learning system that detects policy-violating pins, such as pornography and toxic content, from each pin's image, its text description and user signals such as reports. Tests label sourcing from noisy reports, multimodal modeling, evaluation under class imbalance and moderation serving.

Design a Multimodal Classifier for Unsafe Pins Using Images, Text and User Reports

Company: Pinterest

Role: Machine Learning Engineer

Category: ML System Design

Difficulty: medium

Interview Round: Technical Screen

Pins are images that users save and share, each with a text description, and other users interact with pins, including by reporting the ones they consider inappropriate. Design an ML system that identifies pins violating content policy, such as toxic content and pornography, using the pin's image, its text and user interaction signals such as reports. The discussion is open-ended, but the interviewer probes the ML details closely: expect to justify how you obtain labels, how you combine the image, text and interaction signals, how you train and evaluate the model, and how it is used once deployed. ```hint Signals arrive at different times Consider which signals exist the moment a pin is uploaded and which only accumulate after people have seen it, and what that means for when and how you score a pin. ``` ```hint Where labels come from User reports are plentiful. Think about how reliable a single report is as evidence of a violation, and what else could supply training labels. ``` ### Clarifying Questions - Which policy categories are in scope besides toxic content and pornography, and can one pin violate several at once? - What happens to a flagged pin: removal, reduced distribution, a warning, or human review? How much reviewer capacity is there? - Must a new pin be checked before it is shown to anyone, or is detection shortly after publication acceptable? - For each category, which error costs more: leaving a violating pin up or removing a legitimate one? - Do human-reviewed moderation decisions exist, or only raw user reports? ### What a Strong Answer Covers - Framing as multi-label classification, with per-category thresholds tied to the enforcement action and the review capacity - Label sources: human review versus noisy, exposure-biased reports; hard negatives; label quality - Features from the image, the text (including text inside images), the uploader and user interactions, and how the modalities are fused - Scoring at upload from content, then re-scoring as reports and engagement accumulate - Evaluation under heavy class imbalance on an unbiased sample, with slices and an online prevalence measurement - The serving path, review-queue prioritization, the feedback loop, and monitoring for adversarial drift ### Follow-up Questions - Reports concentrate on popular pins, and some users mass-report content they merely dislike. How do you keep this from distorting training and scoring? - Unflagged pins are never reviewed. How do you estimate how much violating content the system misses? - Uploaders start making small edits to images to evade the classifier. How do you detect and respond to this? - The image is harmless but the text is abusive, or the reverse. How does your model combine conflicting modalities?

Overview: Design a machine learning system that detects policy-violating pins, such as pornography and toxic content, from each pin's image, its text description and user signals such as reports. Tests label sourcing from noisy reports, multimodal modeling, evaluation under class imbalance and moderation serving.

Read the full Pinterest Machine Learning Engineer interview experience this question came from

Pinterest logo
Pinterest
Nov 3, 2026
mediumMachine Learning EngineerTechnical ScreenML System Design
0
0

Pins are images that users save and share, each with a text description, and other users interact with pins, including by reporting the ones they consider inappropriate. Design an ML system that identifies pins violating content policy, such as toxic content and pornography, using the pin's image, its text and user interaction signals such as reports.

The discussion is open-ended, but the interviewer probes the ML details closely: expect to justify how you obtain labels, how you combine the image, text and interaction signals, how you train and evaluate the model, and how it is used once deployed.

Clarifying Questions Guidance

  • Which policy categories are in scope besides toxic content and pornography, and can one pin violate several at once?
  • What happens to a flagged pin: removal, reduced distribution, a warning, or human review? How much reviewer capacity is there?
  • Must a new pin be checked before it is shown to anyone, or is detection shortly after publication acceptable?
  • For each category, which error costs more: leaving a violating pin up or removing a legitimate one?
  • Do human-reviewed moderation decisions exist, or only raw user reports?

What a Strong Answer Covers Guidance

  • Framing as multi-label classification, with per-category thresholds tied to the enforcement action and the review capacity
  • Label sources: human review versus noisy, exposure-biased reports; hard negatives; label quality
  • Features from the image, the text (including text inside images), the uploader and user interactions, and how the modalities are fused
  • Scoring at upload from content, then re-scoring as reports and engagement accumulate
  • Evaluation under heavy class imbalance on an unbiased sample, with slices and an online prevalence measurement
  • The serving path, review-queue prioritization, the feedback loop, and monitoring for adversarial drift

Follow-up Questions Guidance

  • Reports concentrate on popular pins, and some users mass-report content they merely dislike. How do you keep this from distorting training and scoring?
  • Unflagged pins are never reviewed. How do you estimate how much violating content the system misses?
  • Uploaders start making small edits to images to evade the classifier. How do you detect and respond to this?
  • The image is harmless but the text is abusive, or the reverse. How does your model combine conflicting modalities?