Design a production RAG system

Q: Design a production RAG system

This question evaluates a candidate's competency in designing end-to-end Retrieval-Augmented Generation (RAG) systems for enterprise document QA, including architecture choices for ingestion and chunking, embedding strategies, vector indexing and ANN, hybrid retrieval and re-ranking, prompt orchestration, safety/PII controls, multilingual support, scalability, observability, API design, and rollout planning. Commonly asked in ML System Design and information retrieval/NLP interviews to assess the ability to reason about trade-offs between recall, latency, cost, and compliance, it tests both high-level architectural judgment (conceptual understanding) and implementation-level production considerations (practical application).

Q: How do I approach ML System Design interview questions?

ML System Design questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master ml system design interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty ML System Design question, commonly asked during Technical Screen rounds at OpenAI.

Q: What role is this question designed for?

This question is commonly asked for Machine Learning Engineer candidates at OpenAI during technical interviews.

Question

Design a Production RAG System for Enterprise Document QA

Context

You are designing a Retrieval-Augmented Generation (RAG) system to answer questions over large, evolving enterprise document corpora (policies, specs, tickets, wikis, PDFs, spreadsheets, code snippets). The system must support access controls, multilingual content, and strong safety/PII guarantees.

Requirements

Specify the end-to-end architecture and key design choices for:

Ingestion and Chunking

Connectors (file stores, wikis, ticketing systems), parsing (PDF/Office/HTML), and normalization
Chunking strategy: window size, stride, hierarchical metadata, handling of tables and code

Embeddings

Model selection (monolingual vs multilingual), dimensionality, normalization, multi-vector strategy (title/body/table/code)

Indexing and ANN

Vector store choice; ANN algorithm (HNSW/IVF/IVF-PQ) and parameters
Recall/latency/cost trade-offs; sharding and replication

Retrieval Pipeline

Hybrid retrieval (BM25 + dense), filters (ACLs, metadata), time decay, multi-vector fusion
Re-ranking (cross-encoder), multi-stage retrieval, answerability scoring

Prompt Orchestration

Grounding and citations, tool/function calls, context packing and deduplication

Hallucination Mitigation

Attribution checks, coverage thresholds, refusal policy

Caching and Freshness

Query/result/vector caches; invalidation; incremental updates and rebuilds

Multilingual and Safety

Language detection and cross-lingual retrieval; PII redaction and policy enforcement

Scalability, Latency, and SLAs

Capacity planning, concurrency, tail-latency budgets, vector store scaling

Observability and Evaluation

Metrics (retrieval/answer quality), drift monitoring, offline gold sets, synthetic data, online A/B tests

Human Feedback and Cost Controls

Feedback loops, active learning, budget-aware retrieval/generation

Fallback Strategies

When retrieval is weak: clarification, escalation, graceful refusal

API Design and Data Schema

REST/JSON APIs; schemas for documents, chunks, embeddings, and citations

Rollout Plan

Staging, backfill, canary, monitoring, and incident playbooks

Design a production RAG system

Quick Overview

Design a Production RAG System for Enterprise Document QA

Context

Requirements

Solution

Comments (0)