Design LLM search handling long token inputs
Company: OpenAI
Role: Machine Learning Engineer
Category: ML System Design
Difficulty: hard
Interview Round: Onsite
Design an LLM-powered search and question-answering system over a large corpus of documents such as internal wikis, PDFs, logs, contracts, and web pages.
The hard part is that both documents and user queries can be much longer than the LLM context window. A user might paste multi-page logs or a long contract, and the corpus may contain documents hundreds of pages long.
### Constraints & Assumptions
- The system should return grounded natural-language answers using retrieved evidence from the corpus.
- Documents and user queries can exceed the model context window.
- Latency and token cost matter; you cannot send every matching document to the LLM.
- The corpus may include structured and unstructured files, with metadata such as document ID, title, section, timestamp, and permissions.
- Assume embeddings, keyword indexes, and document preprocessing are allowed, but explain how they fit into the system.
### Clarifying Questions to Ask
- What types of questions are most common: fact lookup, summarization, troubleshooting, comparison, or legal-style review?
- What freshness requirements do documents have?
- Are documents permissioned per user or team?
- What context window and latency budget should we design for?
- Do answers need citations or source snippets?
### Part 1 - End-to-End Architecture
Describe ingestion, indexing, retrieval, ranking, prompt construction, answer generation, and feedback logging.
#### What This Part Should Cover
- Parsing and normalizing documents.
- Chunking, metadata storage, vector search, and keyword search.
- Query understanding and retrieval.
- Re-ranking and final LLM answer generation.
- Source citations and access control.
### Part 2 - Handling Long Documents
Explain how documents that exceed the context window are processed and retrieved.
#### What This Part Should Cover
- Chunking with overlap and natural boundaries.
- Hierarchical indexes for document, section, and chunk levels.
- Metadata filters and permission filters.
- Re-ranking and context budgeting.
- Snapshot or summary storage for very long documents.
### Part 3 - Handling Long Queries
Explain how to process long pasted queries without losing important information.
#### What This Part Should Cover
- Query splitting, extraction, summarization, and embedding.
- Separate handling for logs, contracts, and free-form prose.
- Multi-vector retrieval from query snippets.
- Keeping the original user intent visible in the final prompt.
### What a Strong Answer Covers
- A RAG architecture with clear offline and online paths.
- Concrete context-window controls for both corpus documents and user-provided text.
- Retrieval quality tradeoffs, including chunk size, overlap, and re-ranking.
- Latency and cost controls such as caching, smaller preprocessing models, and prompt budgets.
- Grounding, citations, safety, permissions, and evaluation.
### Follow-up Questions
- How would you evaluate whether chunking is causing missed answers?
- How would you support questions that require synthesizing many documents?
- How would you handle permissioned documents in retrieval and caching?
- What would you change if the task were "summarize this entire 200-page document" rather than search?
Overview: Design an LLM search and RAG system that handles long documents and long user queries. Covers chunking, hierarchical retrieval, query compression, context budgeting, citations, latency, and evaluation.