Design Retrieval for a Coding Agent: What to Index, Handling Noise, and Metrics

Design the retrieval system behind a coding agent: an LLM-based assistant that works inside a software code base, reading code, answering questions about it and making edits, and that relies on retrieval to put the right code and documentation into its context window. In this phone screen, the interviewer concentrated on three questions: what should be indexed, how to deal with noise, and how to design metrics that show whether retrieval is working. As in most retrieval-augmented generation (RAG) design rounds, expect to sketch the end-to-end pipeline first and then be pushed deep on individual components. ### Clarifying Questions - Is the corpus one repository, a large monorepo, or many repositories across an organization, and roughly how large is it? - Which programming languages and file types must be supported? - Does the agent call retrieval as a tool it controls, issuing several searches per task, or is context injected automatically before each model call? - How fresh must the index be: the main branch, the user's working branch, or also the edits the agent itself has just made? - What latency budget does one retrieval call have, and how many calls happen during a typical task? - Do access permissions differ between repositories or users? ### Part 1 — The pipeline and what to index Sketch the end-to-end pipeline, from ingestion to the context the agent receives. Then decide what content should be indexed, what should be excluded, how it should be split into retrievable units, and what metadata each unit needs. ```hint Start from the agent's questions List the questions an agent asks while fixing a bug or adding a feature, then ask which artifacts in and around a repository answer each one. ``` #### What This Part Should Cover - An end-to-end pipeline: ingestion, indexing, retrieval, ranking and context assembly - Specific content to include and exclude, with reasons - A splitting strategy that fits source code, and the metadata attached to each unit - How the index stays fresh as code changes ### Part 2 — Handling noise Explain where noise comes from in this system and how you would reduce its impact on what the agent sees. ```hint Noise has several entry points Look separately at noise already in the corpus, noise in the queries the agent sends, and noise in the ranked results; each calls for a different fix. ``` #### What This Part Should Cover - Sources of noise in the corpus, the queries and the results - Filtering and deduplication at indexing time - Ranking and query-side techniques that suppress irrelevant or stale results - What the system does when nothing relevant is found ### Part 3 — Metric design Design the metrics you would use to decide whether retrieval is good enough, to compare two versions of it, and to monitor it in production. ```hint Two layers of quality Separate the quality of the retrieved context from the quality of the agent's final work, and ask where relevance labels could come from without labeling everything by hand. ``` #### What This Part Should Cover - Offline retrieval metrics and how the evaluation set is built - End-to-end metrics that connect retrieval to the agent's success - Online and production signals, including latency, cost and freshness - How to attribute a failure to retrieval rather than to the model ### What a Strong Answer Covers - Scoping questions answered with explicit assumptions about corpus size, freshness and latency - Indexing decisions specific to code rather than generic document chunking - Hybrid retrieval that exploits exact identifiers as well as semantic similarity - Concrete noise controls at indexing, query and ranking time - An evaluation plan with offline and online layers, including where labels come from - Non-functional concerns: latency, cost, index freshness, permissions and monitoring ### Follow-up Questions - How would you choose or fine-tune an embedding model for code, and how would you select positive pairs and hard negatives? - Compare pointwise, pairwise and listwise reranking in this setting. When is a cross-encoder reranker worth its latency? - The agent works over many steps. How should retrieval use what it has already read or edited earlier in the session? - The system must serve thousands of repositories with per-user permissions. What changes in the index and the query path?