Design a chunking strategy for RAG

Quick Overview

This question evaluates understanding of chunking strategies for Retrieval-Augmented Generation systems, testing competencies in information retrieval, embedding and indexing trade-offs, document-structure-aware segmentation, and semantic chunking within the ML System Design and NLP domains.

Design a chunking strategy for RAG

Company: Zillow

Role: Machine Learning Engineer

Category: ML System Design

Difficulty: medium

Interview Round: Technical Screen

You are building a Retrieval-Augmented Generation (RAG) system that uses an LLM plus a vector database. Before creating embeddings and indexing documents, you must **split long documents into chunks**. Describe how you would design the **chunking strategy**. In your answer, discuss: - How you would choose **chunk size** and **overlap** and the trade-offs involved (recall vs. context size, latency, etc.). - How you would use **document structure** (e.g., headings, paragraphs, sections) vs. naive fixed-length splits. - When you might use more advanced methods like **semantic chunking** or dynamic chunk sizes. - How you would evaluate and iterate on your chunking strategy in a real system.

Quick Answer: This question evaluates understanding of chunking strategies for Retrieval-Augmented Generation systems, testing competencies in information retrieval, embedding and indexing trade-offs, document-structure-aware segmentation, and semantic chunking within the ML System Design and NLP domains.

|Home/ML System Design/Zillow
Zillow logo
Zillow
Sep 24, 2025, 12:00 AM
mediumMachine Learning EngineerTechnical ScreenML System Design
9
0

You are building a Retrieval-Augmented Generation (RAG) system that uses an LLM plus a vector database. Before creating embeddings and indexing documents, you must split long documents into chunks.

Describe how you would design the chunking strategy. In your answer, discuss:

  • How you would choose chunk size and overlap and the trade-offs involved (recall vs. context size, latency, etc.).
  • How you would use document structure (e.g., headings, paragraphs, sections) vs. naive fixed-length splits.
  • When you might use more advanced methods like semantic chunking or dynamic chunk sizes.
  • How you would evaluate and iterate on your chunking strategy in a real system.

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...