RAG And Agentic Tool Use

What's being tested

These interviews test whether you can turn an LLM prototype into a measurable, reliable production ML system rather than merely describe embeddings and prompts. You must connect retrieval quality, generation groundedness, agentic tool selection, latency, cost, safety, and continuous improvement into one evaluation loop. Microsoft cares because enterprise assistants operate over changing, permissioned knowledge and must provide evidence-backed answers with predictable behavior. The interviewer is probing for explicit metrics, versioned evaluation data, calibrated judges, failure diagnosis, deployment gates, and monitoring tied to model and retrieval changes.

Core knowledge

• RAG architecture separates query understanding, retrieval, reranking, context construction, generation, and post-generation validation. Treat each stage as observable and replaceable; a fluent answer cannot compensate for low retrieval recall or unauthorized context.

• Hybrid retrieval combines dense vector search with lexical methods such as BM25. Dense embeddings capture semantic similarity, while lexical retrieval preserves exact identifiers, acronyms, error codes, and names; reciprocal-rank fusion is a practical baseline before learned reranking.

• Retrieval evaluation should measure Recall@k, Precision@k, MRR, and nDCG against versioned relevance labels. Also measure answer-level groundedness: retrieved passages must contain sufficient evidence, not merely be topically similar to the query.

• Rerankers such as cross-encoders improve precision by scoring query-document pairs jointly, but add latency and compute cost. A common design retrieves 50–200 candidates cheaply, reranks the top 20–50, then supplies only the best evidence within the context budget.

• Grounded generation requires citation or claim verification, not just a “retrieval succeeded” flag. For claims $c_i$, a simple groundedness estimate is $\frac{1}{n}\sum_i \mathbf{1}[\text{evidence supports }c_i]$; unsupported claims should be abstained, revised, or flagged.

• LLM-as-a-judge is useful for semantic quality but is biased by verbosity, position, and model style. Calibrate it against human labels, use pairwise comparisons where appropriate, report inter-rater agreement, and periodically audit difficult, adversarial, and minority-language examples.

• Evaluation datasets need query, expected answer or claims, relevant evidence, permitted tools, and policy labels. Version them with corpus snapshots, prompts, model IDs, retriever configuration, and judge configuration so regressions are attributable and reproducible.

• Agentic tool use should be modeled as a constrained policy: select a tool, validate arguments, execute with authorization and timeout, inspect the result, and either continue or answer. Use strict schemas, allowlists, idempotency where applicable, bounded steps, and explicit termination conditions.

• Tool-selection metrics include tool-call precision, successful execution rate, argument validity, unnecessary-call rate, task completion, and recovery from tool failures. A correct final answer with an unsafe or unauthorized intermediate call is still a system failure.

• Online monitoring combines quality proxies and operational signals: groundedness, citation coverage, abstention rate, user correction rate, task success, tool errors, token cost, p50/p95/p99 latency, and rate of retrieval-empty or context-overflow events. Segment by tenant, intent, language, and permission scope.

• Improvement loops should isolate one change at a time when possible: corpus/chunking, embedding model, retrieval, prompt, generation model, or tool policy. Run offline regression gates first, shadow or canary traffic next, and roll back automatically when guardrail or quality thresholds regress.

• Distribution shift and freshness are distinct problems: drift in query or document distributions can degrade retrieval, while stale indexes produce outdated answers even when embeddings remain well calibrated. Track freshness, document-version coverage, embedding drift, and performance on newly sampled queries.

Worked example: Design an Evaluation and Improvement Loop for a Production LLM Application

A strong candidate first clarifies the application’s users, supported tasks, knowledge freshness requirements, latency and cost budgets, acceptable risk, and whether answers must include citations or tool actions. They would declare that quality is multi-dimensional: retrieval relevance, groundedness, correctness, completeness, safety, task success, and user-perceived helpfulness. The answer can then be organized around four pillars: a versioned evaluation set, component-level offline metrics, calibrated end-to-end evaluation, and production monitoring with a gated improvement loop. They would compare retrieval labels and Recall@k with claim-level groundedness and judge or human ratings, rather than relying on one overall LLM score. A key tradeoff is judge-model cost versus evaluation coverage: use a cheaper calibrated judge for broad regression testing and humans for a stratified, high-risk sample. They would version prompts, models, corpus snapshots, retriever settings, and tool policies to make failures reproducible. Deployment would use offline thresholds followed by shadow traffic, canaries, and rollback on quality, safety, latency, or cost regressions. If they had more time, they would add slice-based analysis for tenants, languages, intents, adversarial prompts, and newly introduced documents.

A second angle

For Design a RAG system with agentic tools, the same evaluation discipline must cover the trajectory, not only the final response. The candidate should distinguish retrieval failures, wrong-tool selection, invalid arguments, authorization failures, tool-result interpretation errors, and unsupported synthesis. A useful design constrains tool calls with schemas and permissions, then measures success rate, unnecessary calls, recovery rate, step count, latency, and cost alongside answer groundedness. The central framing shifts from “does the model answer well?” to “does the policy complete the task safely and efficiently under bounded autonomy?”

Common pitfalls

Pitfall: Treating retrieval hit rate or cosine similarity as answer quality. Similarity can return semantically related but non-authoritative passages; pair retrieval metrics with evidence sufficiency, claim entailment, citation correctness, and end-to-end task success.

Pitfall: Saying “we monitor user feedback” without defining an evaluation protocol. Explain how explicit ratings, corrections, escalations, sampled traces, and judge scores are labeled, segmented, calibrated, and converted into retraining or rollback decisions.

Pitfall: Hand-waving agent behavior as “the LLM decides which tool to call.” A stronger answer specifies tool schemas, authorization boundaries, retries, timeouts, maximum steps, idempotency, failure recovery, and what happens when evidence or tool results are insufficient.

Connections

The interviewer may pivot to model serving, feature and data versioning, online/offline parity, drift detection, responsible AI, or A/B and canary experimentation. Be ready to explain how evaluation artifacts and telemetry connect to deployment gates and rollback mechanisms.

Further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — foundational RAG architecture and motivation.

• RAGAS: Automated Evaluation of Retrieval Augmented Generation — practical framework for evaluating retrieval and grounded generation.

• Toolformer: Language Models Can Teach Themselves to Use Tools — foundational perspective on tool-use learning and execution traces.