Design an ML search system with RAG
Company: OpenAI
Role: Machine Learning Engineer
Category: ML System Design
Difficulty: hard
Interview Round: Technical Screen
Design an ML-powered enterprise search system using Retrieval-Augmented Generation (RAG).
Context and constraints:
- Corpus: 5M documents (avg 2 KB) from PDFs/web pages/tickets; updates must be searchable within 5 minutes.
- Traffic: 300 QPS; multi-tenant with per-document ACLs.
- SLOs: p95 latency ≤ 1.2 s end-to-end; budget ≤ $0.002 per query.
Sub-questions:
(a) Ingestion and chunking: parsing, deduplication, metadata extraction, embedding generation, chunk-size strategy, versioning, and incremental updates.
(b) Indexing and retrieval: hybrid sparse+vector (BM25 + ANN), metadata filters, tenant isolation, query understanding/reformulation, top-k selection, and cross-encoder reranking.
(c) Generation: prompt design, grounding with citations, constrained decoding, tool usage, streaming responses, and multilingual handling.
(d) Guardrails and safety: hallucination reduction, citation enforcement, out-of-policy refusal, PII/security controls, and ACL-aware retrieval.
(e) Evaluation and monitoring: offline metrics (NDCG@k, recall@k, answer faithfulness), online A/B tests, user feedback loops, drift/latency/cost monitoring.
(f) Architecture and scaling: service decomposition, model hosting/batching, caching, vector store selection, backpressure, failover, and disaster recovery.
(g) Cost and latency calculations: derive per-stage latency/cost, capacity plan for embeddings, ANN index size, and compute requirements. Justify model choices under constraints.
Overview: Design an ML search system with RAG evaluates ML product requirements, data/labeling, modeling, serving architecture, evaluation, monitoring, and trade-offs in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.