Designing Adult Content Detection and Labeling for X (Twitter)

You are a product manager at X (formerly Twitter). Explain how you would build a system that identifies adult content in what people post on the platform, and what the product should do with that content once it is found. Cover who the system protects and serves, how detection would work at the volume X handles, what happens to a post after it is flagged, and how you would know the system is working. ```hint Detection is not the same as removal Before picking models, decide what should happen to adult content the platform allows, to content that is illegal, and to content that only looks adult. Each outcome needs a different level of accuracy and a different speed. ``` ### Clarifying Questions - Is the aim to take adult content down, or to label and restrict it so that minors and people who have not opted in do not see it while consenting adults still can? - Which formats are in scope: images and video only, or also text, links, live video and profile or header images? - Is clearly illegal material, such as sexual content involving minors or intimate images shared without consent, part of this system or handled by a separate team and pipeline? - For the first version, which error worries us more: explicit media reaching people who never chose to see it, or legitimate posts such as art, health education or breastfeeding photos being hidden? ### What a Strong Answer Covers - A goal and policy taxonomy that separate permitted-but-sensitive adult content, prohibited content and look-alikes, with a different action for each. - The stakeholders and their competing needs: viewers who have not opted in, minors, creators who post adult content within the rules, advertisers, moderators and regulators. - A layered detection design (hash matching, media and text classifiers, account-level signals, creator self-labels, user reports) with confidence thresholds that route each item to automatic action or human review. - The product response after detection (labels, blurring, age gates, distribution limits, appeals) and a feedback loop that turns outcomes into training data. - Success metrics that weigh missed content against wrongly labeled posts, measured per surface, plus guardrails and risks such as evasion, bias and moderator well-being. ### Follow-up Questions - A network of accounts starts evading the classifier by cropping images, adding overlays and moving explicit media behind links in replies. How do you detect and respond? - Creators who post adult content within the rules complain that their ordinary posts are being labeled and their reach has collapsed. How do you decide whether to loosen thresholds? - How would you estimate how much unlabeled adult content people still see, when by definition the system cannot count what it missed?