Design a chunking strategy for RAG
Company: Zillow
Role: Machine Learning Engineer
Category: ML System Design
Difficulty: medium
Interview Round: Technical Screen
You are building a Retrieval-Augmented Generation (RAG) system that uses an LLM plus a vector database. Before creating embeddings and indexing documents, you must **split long documents into chunks**.
Describe how you would design the **chunking strategy**. In your answer, discuss:
- How you would choose **chunk size** and **overlap** and the trade-offs involved (recall vs. context size, latency, etc.).
- How you would use **document structure** (e.g., headings, paragraphs, sections) vs. naive fixed-length splits.
- When you might use more advanced methods like **semantic chunking** or dynamic chunk sizes.
- How you would evaluate and iterate on your chunking strategy in a real system.
Quick Answer: This question evaluates understanding of chunking strategies for Retrieval-Augmented Generation systems, testing competencies in information retrieval, embedding and indexing trade-offs, document-structure-aware segmentation, and semantic chunking within the ML System Design and NLP domains.