## Scenario Design an AI chatbot that answers user questions from an approved knowledge corpus while supporting multi-turn conversation. The system should provide responsive answers, avoid leaking content the user is not authorized to read, and make uncertain or unsupported answers visible rather than presenting them as facts. ### Constraints & Assumptions - Documents can be created, updated, deleted, and assigned access-control rules. - A conversation may refer back to earlier turns, but context growth must be bounded. - Retrieved text and user input are untrusted data, not instructions that can override system policy. - The design must support model timeouts, retrieval failures, rate limits, and safe retries. - Exact traffic, latency, model, and retention targets are not supplied; state reasonable assumptions and identify where they affect the design. ### Clarifying Questions to Ask - Is the bot limited to retrieval-grounded answers, or may it answer general questions? - What identity and document-permission system is authoritative? - What latency, availability, cost, and answer-quality targets matter? - Must answers include citations, and what should happen when evidence is weak or conflicting? - How long may messages, retrieved passages, and feedback be retained? ```hint Put authorization before generation Filter candidates using the caller's current permissions before any passage can enter the model context. ``` ```hint Treat quality as an observable pipeline Record retrieval and generation decisions in a privacy-safe trace so failures can be attributed to ingestion, retrieval, prompting, or the model. ``` ### What a Strong Answer Covers - Ingestion, normalization, chunking, versioning, indexing, and deletion propagation. - A request path with authentication, query rewriting, hybrid retrieval, permission filtering, reranking, context assembly, generation, and citation validation. - Conversation-state summarization that preserves relevant facts without allowing unbounded prompts. - Prompt-injection defenses and a clear boundary between data and trusted instructions. - Grounded refusal or uncertainty behavior when evidence is missing. - Caching, streaming, backpressure, fallbacks, idempotency, and cost controls. - Offline evaluation sets, online feedback, safety metrics, and end-to-end observability. ### Follow-up Questions 1. How do edited or revoked documents disappear from answers and caches quickly? 2. How would you measure whether a bad answer came from retrieval or generation? 3. What prevents a document from instructing the model to reveal another user's context? 4. How would the architecture change for tool calls that can mutate external systems?