Designing Layered Safeguards That Stop Harmful Uploads on a Social Platform

You are a product manager at a social media platform. How would you keep "bad" content from being uploaded in the first place? ```hint Not all bad content is alike Separate the kinds of bad content by how much harm they cause and how reliably they can be detected, then look at what can happen before, during and just after an upload, not only at the moment a file arrives. ``` ### Clarifying Questions - What counts as "bad" here: illegal material such as child sexual abuse imagery, policy violations such as hate speech or graphic violence, misinformation, spam and scams, or content that is allowed but low quality? - Which platform and formats are in scope: photos, short videos, long videos, text posts or livestreams? - For the most severe categories, should we accept blocking some legitimate posts to get as close to zero exposure as possible, or balance safety against friction for creators? - Is the scope strictly stopping content at upload, or may I also limit the reach of content that has already been posted? ### What a Strong Answer Covers - A taxonomy of bad content that separates categories by severity and by how detectable they are - Who posts bad content and why: creators unaware of the rules, people chasing engagement or money, and organized bad actors - Controls at each stage (account creation, composing, upload and early distribution), with human review where machines are unsure - Trade-offs between harm prevented, wrongful removals, upload latency and review cost - Metrics such as prevalence and proactive detection, with guardrails for wrongful removals and creator experience ### Follow-up Questions - Livestreams cannot be scanned before they air. How would your approach change for them? - A new harmful trend spreads faster than you can label training data for it. What would you do in the first two days? - How would you decide the confidence level at which an upload is blocked automatically rather than sent to human review?