Keeping LLM Output Safe: Dual-Use Requests, Jailbreaks and Layered Defenses

Generative AI models can produce content that is harmful or unsafe. What makes preventing this difficult, and how would you address each of those difficulties? ```hint Think about the legitimate user and the attacker Consider why a rule that blocks a dangerous request can also block a legitimate one, and what a determined user does once they know a protection exists. ``` ### Clarifying Questions - Which deployment should I focus on: a consumer chat assistant, an API that businesses build their own products on, or both? - Which kinds of harm are in scope: help toward serious violence or weapons, self-harm, harassment and hate, privacy leaks, or confidently wrong advice in areas like medicine? - Is this about text only, or also images, code, and agents that take actions? - Should one standard apply to every customer, or can a business adjust what is allowed for its own setting, such as a medical or security-research platform? ### What a Strong Answer Covers - Why the problem is hard: context-dependent and contested definitions of harm, dual-use information, an open-ended output space, adaptive adversaries, and new modalities. - The tension between preventing harm and refusing legitimate requests, and how to measure both sides. - A layered set of mitigations across policy, training, runtime checks, product design and abuse enforcement, rather than a single filter. - Evaluation and red teaming before release, plus monitoring and incident response afterward. - Prioritizing by severity, so the most catastrophic risks are handled differently from offensive but low-harm content. ### Follow-up Questions - A new jailbreak spreads online and works against your production model. Walk through how you respond. - Users complain that the model refuses harmless medical and security questions. How do you reduce refusals without raising risk? - How would your approach change for a model whose weights are released openly?