Design the Security Architecture for an LLM Chatbot Product

Design the security for a company's chatbot product: a production chatbot that holds conversations with people outside the company. Assume the chatbot is built on a large language model (LLM). Cover how you would protect the chatbot, the data it can see, the systems it can reach and the people who talk to it. Start by establishing what the chatbot does and who uses it, then build a threat model, then design the controls and the architecture that enforces them. ```hint Start from attackers and assets List who could try to misuse the chatbot (the people chatting with it, the content it reads, other customers of the product, insiders) and what each of them could reach, before picking any control. ``` ```hint Separate what the model says from what the system does Decide which decisions you would never leave to the language model alone, and where in the request path those decisions should be enforced instead. ``` ### Clarifying Questions - Is the chatbot run for many business customers, each exposing it to its own users (multi-tenant), or for a single organization? - Does it only answer questions from a knowledge base, or can it also take actions in other systems through tools, such as looking up or changing a user's account? - Are the people chatting anonymous visitors, signed-in users of the customer's product, or both? - What sensitive data can appear in conversations or in the knowledge it reads: personal data, payment details, health information? - Is the language model hosted by the company or called through a third-party API? - Which regulatory, data-residency and retention requirements apply? ### What a Strong Answer Covers - A threat model naming attackers, assets and entry points specific to an LLM chatbot, including direct prompt injection by users and indirect injection through content the bot reads - Authentication of the people chatting, and authorization of every data access and tool call outside the model, with the requesting user's permissions - Isolation between customers' data, retrieval indexes, conversation histories and configuration - Handling of sensitive data in prompts, logs and calls to a model provider: minimization, redaction, encryption and retention - Output-side controls against leaking system instructions, secrets or other users' data, and against unsafe responses - Abuse and cost controls, audit logging, monitoring and an incident response path - Trade-offs between protection, latency, cost and how useful the bot remains ### Follow-up Questions - A document in a customer's knowledge base contains hidden instructions telling the bot to reveal other users' data. Walk through what stops that attack at each layer. - Suppose the bot gets a tool that moves money, such as issuing refunds. How do you stop a user from talking it into refunds they are not entitled to? - How would you detect in production that your controls are being bypassed, and what would you do in the first hour after detection? - A customer requires that its users' conversations stay in one region and are never used to train models. What changes in the design?