Design a Multi-Tenant Enterprise RAG Assistant: Hybrid Retrieval, Reranking, ACLs

Design a retrieval-augmented generation (RAG) assistant for enterprise customers, in the style of ChatGPT Enterprise: employees ask questions in natural language, and the assistant answers using their company's internal documents (wikis, tickets, shared drives, chat exports), citing its sources. Many companies (tenants) share the platform. Cover **embeddings**, **hybrid retrieval** that combines keyword and vector search, **reranking**, and **multi-tenancy**, from document ingestion through to the generated answer. ```hint Permissions are part of retrieval Two employees of the same company may be allowed to see different documents; decide at which step access control must be applied so that a forbidden document can never reach the model. ``` ```hint Why two kinds of search Think of queries that vector search handles badly, such as exact error codes, product names or ticket IDs, and queries that keyword search handles badly, and how the results of both can be merged. ``` ### Clarifying Questions - How many tenants, documents per tenant and queries per second are expected, and how large are the largest tenants? - Which sources must be connected, and how fresh must the index be after a document changes? - Do document permissions have to be enforced per user, following the source systems' access controls? - What latency is acceptable for an answer, and is streaming expected? - Are there data-residency or isolation requirements, such as a dedicated index per tenant or no cross-tenant model training? - How will answer quality be measured? ### What a Strong Answer Covers - An ingestion pipeline: connectors, parsing, chunking, metadata and permission capture, incremental updates and deletions - Embedding model choice, index design, and the cost of re-embedding when the model changes - Hybrid retrieval and score fusion, then reranking, with their latency and quality trade-offs - Multi-tenant isolation of data and indexes, permission-aware retrieval, and noisy-neighbor control - Grounded generation with citations, handling of insufficient context, and prompt-injection risks from documents - Offline and online evaluation of retrieval and answers, and monitoring ### Follow-up Questions - How would you fine-tune the embedding model for a customer's domain, and what training data and loss would you use? - How could you mine new training data for retrieval from a large unlabeled document corpus? - A document is deleted in the source system. How quickly and how reliably does it disappear from answers? - How would you reduce latency and cost for the most frequently asked questions?