The interviewer wants two things at once: a coherent end-to-end RAG design, and a credible evaluation story for *every* stage. The trap is to over-index on architecture and hand-wave evaluation. I keep architecture tight and make evaluation a first-class deliverable — for each component I state what can go wrong, the metric that detects it, and how I get the labels. ## 1) Clarify scope before designing A 5-minute scoping pass changes the entire design, so I lead with it: - **Corpus**: size ($10$ docs vs $10^6$ chunks), formats (clean Markdown vs scanned PDFs with tables/images), languages, and *update frequency* (the requirements call out frequent changes — so freshness is a hard constraint, not a nice-to-have). - **Query mix**: single-fact lookup vs multi-hop reasoning vs summarization vs "list all X." This decides chunk size, top-$k$, and whether I need iterative retrieval. - **Latency/cost budget**: e.g. p95 $< 2\text{s}$ end-to-end. This bounds how many LLM calls I can chain (query rewrite, rerank, generation, self-check). - **Access control**: per-user/per-tenant ACLs? If yes, retrieval *must* filter by entitlement — a leak here is a security incident, not a quality bug. - **Output contract**: free text vs structured JSON; mandatory citations; abstention allowed? **Working assumptions** for the rest of the answer: internal docs at $\sim 10^6$ chunks, mixed PDF/HTML/Markdown, daily-to-streaming updates, mandatory citations, interactive latency, per-user ACLs. --- ## 2) End-to-end architecture and data flow Two planes, decoupled so ingestion never blocks queries. **Offline / streaming indexing plane** ``` Sources (S3 / Drive / Confluence / Git) → Ingestion + change detection (content hash, watermark) → Parse & normalize (layout-aware; preserve headings, tables, page #) → Chunk (structure-aware) + attach metadata (doc_id, section, ts, ACL, url) → Embed (chunk vector; optional title/summary vector) → Upsert into Vector index (HNSW) + Lexical index (BM25) + doc store ``` **Online query plane** ``` User query + identity → AuthZ: resolve entitlements → ACL filter → Query understanding: rewrite / expand / extract metadata filters → Hybrid retrieve: vector + BM25 → fuse (RRF) → top-N candidates → Rerank (cross-encoder) → top-k → Context assembly: dedupe, order, compress, attach citations → Generate: grounded answer + citations, with abstention rule → Post-process: PII/safety filter, citation validation → Response + log everything (query, doc_ids, scores, tokens, latency) ``` Everything is logged with IDs so any answer is fully replayable for debugging and offline eval. The two planes share only the index, which lets me re-embed or re-chunk the whole corpus (a migration) without touching serving. --- ## 3) Ingestion, parsing, chunking **Parsing.** Format-specific, layout-aware extraction. The hard cases are PDFs: I preserve page numbers, reading order, and headings, and handle tables explicitly (linearize to Markdown or keep a structured representation) rather than letting them collapse into word soup. I strip repeated headers/footers/boilerplate, which otherwise pollute embeddings and inflate duplicate retrieval. Scanned PDFs route through OCR. **Chunking.** Structure-aware beats fixed-size. I split on document structure (sections/headings) first, then pack to a target token budget (a few hundred tokens) with modest overlap so a sentence cut at a boundary is still recoverable. Each chunk carries metadata: `doc_id`, `section_title`, `updated_at`, `acl_tags`, `source_url`, `page`. I prepend the section/title path into the embedded text so an isolated chunk keeps its context (`Billing > Refunds > Eligibility: ...`). **Pitfalls to name:** too-small chunks lose context and split answers across units; too-large chunks dilute the embedding and waste prompt budget; PDF extraction noise; near-duplicate boilerplate; tables that lose row/column structure. **Evaluating ingestion/chunking** (usually unmeasured — I make it explicit): - *Coverage/integrity*: parser error rate, fraction of pages with zero extracted text (signals scanned docs needing OCR), table-extraction success on a labeled set of hard docs. - *Chunk health*: length distribution, overlap %, near-duplicate rate (embedding cosine or MinHash). - *Answerability proxy*: from a held-out set of `(question, gold answer span)`, what fraction of gold spans fall entirely inside a single chunk? Low values mean chunking is splitting answers — the most actionable ingestion metric, because an answer bisected across a boundary is much harder to surface cleanly (overlap or multi-chunk assembly can sometimes stitch it back, but it's fragile). - *Regression suite*: a fixed set of "known-hard" documents (multi-column PDFs, big tables) that re-runs on every parser/chunker change. --- ## 4) Embeddings and indexing **Embeddings.** A strong general-purpose text embedding model is the baseline. If the corpus is jargon-heavy (legal, medical, internal acronyms), I'd consider fine-tuning/adapting the embedder on in-domain `(query, relevant-chunk)` pairs — but only after measuring that the off-the-shelf retriever actually underperforms, since adaptation adds a training and refresh burden. I **version the embedding model**; a model change requires re-embedding the whole corpus, and queries must be embedded with the same version as the index. **Index.** Approximate-nearest-neighbor (HNSW for low-latency recall, or IVF/PQ if memory-bound) with **metadata filtering** for ACL/tenant/doc-type/time. I pair it with a **lexical (BM25) index** for hybrid search — vectors handle paraphrase and semantics; BM25 handles exact matches that embeddings famously miss (error codes, IDs, product names, rare tokens). Fuse the two with **Reciprocal Rank Fusion** so I don't have to calibrate score scales across the two systems. **Freshness (a hard requirement).** Change detection by content hash + source watermark. On a changed doc: re-parse, re-chunk, re-embed only affected chunks, upsert by stable chunk key, and **tombstone** deleted chunks so stale content can never be retrieved. Track end-to-end index lag (source change → searchable) as an SLO. **Evaluating retrieval (the metric that matters most in RAG).** Generation quality is capped by what retrieval surfaces — if the answer isn't in context, the model either hallucinates or abstains. So I build a labeled set of `(query → relevant chunk/doc IDs)` and measure (the same labeled set scores both this index and the §5 retrieve-and-rerank stage): - **Hit-rate@k (success@k)**: fraction of queries where at least one truly-relevant chunk appears in the top-$k$ handed to the generator. This is the ceiling on answerable questions; for single-answer lookups it's the metric I watch most. - **Recall@k**: $\lvert\text{relevant} \cap \text{top-}k\rvert / \lvert\text{relevant}\rvert$ — what share of *all* relevant chunks made the cut. Matters for "list all / aggregate" queries where one hit isn't enough. - **nDCG@k / MRR**: is the relevant chunk ranked near the top? Order matters because reranking and prompt-budget truncation are sensitive to it. - **Context precision**: fraction of retrieved chunks that are actually relevant (noise crowds out signal and raises cost). I **slice** every metric by doc type, query type, tenant, and freshness — aggregate recall hides that, say, table-heavy docs are failing. **Getting labels without a gold set:** generate synthetic `(question, answer, source-chunk)` triples by prompting an LLM over each chunk, with human spot-checks to control quality; mine click/feedback logs as weak labels; and have annotators judge relevance on the top results of real queries. Synthetic eval is directionally useful and cheap to refresh as the corpus changes, but I'd never ship purely on it without a human-verified slice. --- ## 5) Retrieval, query understanding, and reranking **Query understanding.** Rewrite the conversational question into a retrieval-friendly query: expand acronyms, resolve coreference from chat history, and extract structured filters ("docs from 2024" → `updated_at >= ...`). For recall on hard queries, **multi-query expansion** (generate 3–5 paraphrases, retrieve each, union) helps — at the cost of latency, so it's a tunable knob. **Two-stage retrieve → rerank.** Retrieve a wide net (top-$N$, e.g. 50–100) cheaply with hybrid search, then **rerank** with a cross-encoder that jointly attends to `(query, chunk)` and produces a far more precise relevance score than the bi-encoder used for ANN. Keep the reranked top-$k$ (e.g. 5–10) for the prompt. The bi-encoder optimizes recall and scale; the cross-encoder optimizes precision on a small candidate set — using both is how you get high recall *and* high precision within budget. **Context assembly.** Dedupe near-identical chunks; for broad questions, diversify across sources rather than returning ten paraphrases of the same paragraph; optionally **compress** (extractive sentence selection or per-chunk LLM summarization) to fit token budget while preserving the citable spans. Order matters — place the highest-scored context where the model attends best, and keep stable citation anchors so the generator can reference passages precisely. **Evaluating reranking.** Compare ranking metrics (nDCG@10, MRR, context precision) with reranker **on vs off** on the same retrieval candidates — the only honest measure of its marginal value. Track the latency/cost it adds, since a reranker that buys +2% nDCG for +300ms may not be worth it. Error-analyze its biases (over-weighting long passages or surface keyword overlap). --- ## 6) Generation, grounding, guardrails, and fallback **Prompt/context contract.** System instruction: answer *only* from the provided context; cite the specific source for each claim; if the context doesn't support an answer, say so rather than guessing. I pass each passage with a stable citation handle (e.g. `[1]` → `doc_id#section`, url) and require inline citation markers in the output, which makes faithfulness checkable. **Grounding / anti-hallucination.** - **Abstention on weak retrieval**: if the top reranker score (or fused confidence) is below a calibrated threshold, route to a fallback rather than generating — this is the single biggest hallucination lever. I tune the threshold on the precision/recall tradeoff of abstention (see §7). - **Clarification**: if the query is ambiguous/underspecified, ask a clarifying question instead of answering. - **Self-check (budget permitting)**: a lightweight verification pass that checks whether each answer sentence is entailed by a cited chunk, and drops/flags unsupported sentences. **Fallback ladder when retrieval is weak:** (1) abstain with an honest "I couldn't find this in the docs"; (2) offer the closest retrieved sources for the user to inspect; (3) ask a clarifying question; (4) route to human/ticket if it's a support setting. Never silently fall back to the model's parametric knowledge for a private-corpus product — that's exactly the ungrounded answer the requirements forbid. **Guardrails.** ACL enforced at retrieval time *and* re-verified before the answer is returned (defense in depth — a chunk the user can't see must never appear in a citation). PII redaction and safety/policy filters on input and output. Citation validation: drop any citation that doesn't resolve to a retrieved, entitled chunk. **Evaluating generation.** I separate three orthogonal axes — a fluent answer can be wrong, and a correct answer can be ungrounded: 1. **Answer correctness** — does it actually answer the question (vs gold answers)? 2. **Faithfulness / groundedness** — is every claim supported by the cited context (the RAG-specific failure)? 3. **Citation quality** — do the citations actually back the claims and point to real, retrieved sources? Methods, in order of trust: - **Human rubric** (gold standard early on): correctness, completeness, groundedness, citation accuracy, readability — on a sampled, sliced set. - **LLM-as-judge** for scale, but only after **calibrating it against human labels** (measure judge-vs-human agreement; recalibrate periodically). Judges have known biases (length, position, self-preference), so I use pairwise comparison and reference-grounded prompts, not bare 1–10 scores. - **Automated faithfulness checks**: require each answer sentence to map to ≥1 cited chunk; run an entailment/NLI or contradiction check between answer sentences and their cited context. Imperfect, but cheap and good for regression gating. --- ## 7) End-to-end evaluation Component metrics can all look green while the product is bad, so I also measure the whole pipeline on a held-out **eval set of real-ish queries**: - **Task success rate** (human- or calibrated-judge-graded end-to-end correctness + groundedness). - **Abstention calibration**: abstention rate vs accuracy-when-answered. Too eager to abstain → useless; too eager to answer → hallucinations. The right operating point is a **product decision**; I report the full tradeoff curve, not one number. - **User outcomes** (online): thumbs up/down, "answer was helpful," citation click-through, and for support use cases, deflection/resolution rate. - **Latency** p50/p95 and **cost per query**, broken out by stage (retrieval, rerank, generation) so I know what to optimize. Crucially, I run this eval set as a **regression gate in CI** for any change to chunking, embedder, $k$, reranker, prompt, or model — so I catch a regression from a "harmless" prompt tweak before users do. --- ## 8) Online monitoring and the improvement loop **Log everything** per query: raw + rewritten query, retrieved doc IDs and scores, rerank scores, final prompt, output, citations, latency per stage, and user feedback. This is both the debugger and the source of future eval data. **Monitor for drift and degradation:** - Retrieval-score and abstention-rate distributions over time (a sudden shift flags a bad index update or embedder regression). - Index freshness lag SLO. - Query distribution drift (new topics the corpus doesn't cover) and embedding distribution shift. - Cost/latency SLOs with alerting. **Closed loop:** mine thumbs-down and abstentions for failure clusters → root-cause to a stage → fix at that stage (chunking, $k$, reranker threshold, synonyms/metadata, prompt) → **add the case to the eval set** so the fix is regression-protected. The eval set is a living asset that grows with every real failure. --- ## 9) Edge cases worth naming - **Conflicting sources**: prefer the most recent/authoritative; surface multiple citations and let the user see the conflict rather than silently picking one. - **Multi-hop questions**: iterative/agentic retrieval — retrieve, draft a sub-question, retrieve again — instead of one shot. - **"List all / aggregate" queries**: pure top-$k$ similarity under-retrieves; needs higher recall, metadata filtering, or a structured query path. - **Very long docs**: hierarchical retrieval (doc summary → section → chunk) to avoid drowning in one document. - **Access control**: per-user ACL filtering is non-negotiable and tested explicitly. --- ## Addressing the follow-up questions **Freshness — edit vs deletion.** Change detection runs on a content hash and source watermark per source object. On an **edit**: re-parse and re-chunk the doc, diff against existing chunk keys, re-embed only chunks whose text changed, and `upsert` them by stable chunk key (so unchanged chunks keep their vectors). On a **deletion** (or a chunk that no longer exists after an edit): I **tombstone** the chunk key immediately — a delete/soft-delete in both the vector and lexical indexes — and only then garbage-collect. Because the online plane filters out tombstoned keys, stale content can't be retrieved even if physical GC lags. The guarantee is monitored via the index-lag SLO and a periodic reconciliation job that compares source object IDs against indexed chunk keys to catch missed deletes. **Root-causing one bad answer.** Because every query is logged with the retrieved doc IDs, scores, rerank scores, final prompt, and output, I can replay it offline and localize the failure to one of three stages: (1) **chunk missing from retrieval** — the gold chunk isn't in the top-$N$ candidates at all → a chunking or embedding/index problem (fix chunk size/overlap, hybrid weighting, synonyms/metadata); (2) **mis-ranked** — the gold chunk is in the candidate pool but the reranker dropped it below top-$k$ → a reranker problem (threshold, model, training data); (3) **present-but-ignored** — the gold chunk *was* in the prompt but the model didn't use it or contradicted it → a generation/grounding problem (prompt contract, self-check, context ordering/compression). The discriminator is simply "was the gold chunk in candidates? in the prompt?", which the logs answer directly. **Evaluating the judge.** I treat the LLM-as-judge as a model to validate, not a source of truth. I hold out a human-labeled slice and measure judge-vs-human **agreement** (e.g. accuracy/correlation, and per-class agreement for faithful vs unfaithful). I guard against known biases — length bias (preferring longer answers), position bias (order of A/B in pairwise prompts, mitigated by swapping and averaging), and self-preference (a judge favoring outputs from its own family). I prefer **pairwise, reference-grounded** prompts over bare 1–10 scores, recalibrate periodically (the corpus and answer distribution drift), and only let the judge gate regressions on axes where its human agreement is high; low-agreement axes stay human-reviewed. **Multi-hop and "list-all" queries.** A single top-$k$ similarity pass structurally under-retrieves these. For **multi-hop**, I use **iterative/agentic retrieval**: retrieve, let the model draft the next sub-question, retrieve again, and accumulate context across hops. For **"list all / aggregate"**, recall (not hit-rate) is the binding metric — one relevant hit isn't enough — so I raise $N$, lean on **metadata filtering** and a **structured query path** (e.g. translate the request into a filter over the doc store) rather than pure vector similarity, and diversify context across sources during assembly. Both cases are sliced separately in eval, because their failure profile and the metric that catches them differ from single-fact lookup. --- ## Why this structure answers the question Each stage has a clear interface, a named failure mode, and a metric that detects it — with **retrieval recall as the upstream ceiling** and **groundedness as the RAG-specific generation metric** — all backed by a regression-gated eval set and an online loop that feeds real failures back into that set. That directly delivers what was asked: walk every component *and* show how to evaluate each step.