Generative AI System Design Interview Questions: RAG, Agents, Evals, and Guardrails

A generative AI system design interview is rarely won by drawing a vector database and placing an LLM box beside it. The difficult part is explaining how a probabilistic model becomes one component inside a system that must retrieve the right evidence, take only authorized actions, detect regressions, and recover when it is wrong.

This guide gives you a reusable answer structure and the most important RAG, agent, evaluation, and guardrail questions to practice. You can also use PracHub's ML system design interview questions to rehearse complete prompts with written solutions before your interview.

Generative AI system design interview covering RAG agents evals and guardrails

Quick answer: what interviewers are really testing

A strong answer treats the model as a useful but untrusted decision-making component. You should define the product objective, identify the highest-cost failure, separate deterministic controls from model judgment, and attach an evaluation plan to every important design choice.

Area	What a strong candidate explains
Problem framing	The user, success metric, unacceptable failure, and when the system should abstain.
RAG	Ingestion, retrieval, reranking, grounding, citations, permissions, and component-level quality.
Agents	Why autonomy is needed, how tools are authorized, and how execution is bounded and recoverable.
Evals	Versioned tasks, production-like slices, mixed graders, repeated trials, and release gates.
Guardrails	Layered defenses around untrusted input, model output, tool calls, data access, and human approval.
Operations	Latency, cost, observability, fallbacks, incident response, and safe rollback.

The interview is therefore not a memorization test. It is a technical judgment test about where generative AI is appropriate, where ordinary software should remain in control, and how you will know the complete product is working.

A reusable framework for any GenAI design prompt

Begin with objective and risk. Ask who uses the system, what action they are trying to complete, which errors are reversible, and which errors could expose data, move money, or damage trust. This immediately determines whether you need a simple retrieval workflow, a fixed tool chain, or a more autonomous agent.

Then walk through six layers: data and context, model boundary, orchestration, evaluation, serving, and recovery. State what enters each layer, what it is allowed to produce, how its output is validated, and what happens when the validation fails.

Generative AI system design control loop from scope and retrieval to evaluation monitoring and recovery

Keep one principle visible throughout the answer: model output proposes; deterministic systems authorize and execute. The LLM may select a candidate tool or draft parameters, but an application service should validate the schema, check the user's permissions, enforce policy, assign an idempotency key, and record the result.

RAG system design interview questions

How would you design the ingestion and retrieval pipeline?

Separate the offline ingestion path from the online serving path. Ingestion should parse documents, preserve metadata and access-control labels, create chunks, generate embeddings, and version the searchable index. Serving should normalize the query, apply tenant and permission filters, retrieve candidates, rerank them, build a bounded context, and generate an answer with citations.

Explain how updates and deletes propagate. A production design needs document lineage, an index version, a tombstone or deletion workflow, and a strategy for switching traffic between old and new indexes without mixing incompatible embeddings.

How do you measure whether RAG is working?

Do not use answer quality as the only metric. Measure parsing coverage and freshness at ingestion; recall@k, precision, or NDCG for retrieval; reranker quality; citation correctness and groundedness for generation; and task completion, latency, and cost for the end-to-end product.

Include a no-answer set. If the evidence is missing or contradictory, the correct behavior may be to abstain, ask a clarifying question, or escalate rather than produce a fluent guess. Also test permission boundaries so a relevant but unauthorized document never reaches the prompt.

Agent system design interview questions

When should you use an agent instead of a workflow?

Use the simplest architecture that meets the requirement. A fixed workflow is easier to test when the steps are known; an agent is justified when the task requires dynamic planning, tool choice, or adaptation to intermediate results. The extra flexibility costs more, takes longer, and introduces a larger failure surface.

Represent execution as durable state rather than a conversation trapped in memory. Store the goal, current step, tool results, approvals, retry count, and terminal status. That lets workers resume after failure, operators inspect a trace, and the system avoid repeating an irreversible action.

How do you make tool use safe and reliable?

Expose narrow, typed tools through a gateway. Validate arguments, authenticate as the user or service identity, apply least privilege, enforce rate and spend limits, and separate read-only tools from high-impact writes. Require confirmation for actions such as sending a message, changing access, or issuing a refund.

Every write should define retry behavior and idempotency. Add maximum steps, elapsed-time limits, token and cost budgets, loop detection, and a human escalation path. A confident answer also explains what happens when a tool succeeds but the worker times out before recording the result.

Evaluation system design interview questions

How would you build an evaluation harness?

Start with a versioned task set sampled from realistic traffic and known failure modes. Slice it by user type, language, document quality, ambiguity, tool path, and risk level so a high average score cannot hide a dangerous regression in one group.

Use the cheapest reliable grader for each assertion. Deterministic checks can verify schemas, citations, tool arguments, database state, latency, and cost. Model graders can score qualities such as helpfulness or groundedness, but their rubrics should be specific and regularly calibrated against expert human judgment.

What changes for agent evaluations?

Grade both the final result and the trajectory. An agent can reach a plausible answer while calling an unauthorized tool, wasting twenty steps, or modifying the wrong record. Capture the full trace, required and forbidden tool calls, state transitions, number of turns, retries, and side effects.

Because agent behavior varies between runs, repeat important tests and report success rates rather than treating one pass as proof. Make the suite a release gate for prompt, model, retrieval, tool, and policy changes, then use canaries and production feedback to add new cases.

Guardrail and safety interview questions

How do you defend against prompt injection?

Draw trust boundaries first. User messages, retrieved documents, web pages, emails, and tool responses are untrusted data even when they look authoritative. Do not place them into privileged instructions, and do not let free-form text flow directly into a sensitive tool.

Use structured outputs, schema validation, content isolation, least-privilege credentials, sandboxed execution, destination allowlists, and explicit approvals. Filters and model-based guardrails are useful layers, but neither RAG nor a stronger system prompt guarantees that an indirect injection will be ignored.

What should happen after a safety failure?

Design for containment and learning. Record the model, prompt, policy, index, tool, and trace versions; revoke or narrow affected credentials; preserve evidence; block the failing action; and route the case for review. Feed the incident back into red-team tests and the regression suite before restoring broader access.

For high-consequence workflows, explain the human control point and the blast radius. A useful guardrail answer names both the protection and its residual risk instead of claiming that one classifier makes the system safe.

Serving, latency, cost, and a worked example

Assign a latency and cost budget to each stage: query processing, retrieval, reranking, generation, and tools. Useful levers include semantic caches with permission-aware keys, smaller models for routing or extraction, parallel independent calls, streaming, bounded context, timeouts, and graceful degradation to search results or human support.

Suppose the prompt is design an enterprise support agent. Use RAG to answer policy questions with current, access-controlled documents and citations. Use an agent only when the request requires multiple steps, such as checking an order, confirming eligibility, and proposing a refund.

The refund tool should receive validated order ID, reason, amount, user identity, approval status, and idempotency key. Evals should test retrieval quality, policy compliance, communication, tool sequence, final database state, and repeated-run reliability. Guardrails should block untrusted documents from changing tool policy, require approval above a threshold, and escalate when evidence conflicts.

Hard follow-ups and common mistakes

Expect the interviewer to remove a comfortable assumption. They may ask what happens when retrieval recall drops, the model judge prefers verbose answers, a retrieved document contains malicious instructions, a tool succeeds before a retry, one tenant's data appears relevant to another tenant, or model latency doubles during a traffic spike.

Weak answers usually over-agent the problem, treat a vector database as the entire RAG design, use prompts as an authorization boundary, trust one aggregate eval score, omit abstention, or lack trace replay and rollback. A stronger answer localizes the failure, names a measurable signal, and changes the design without rebuilding every component.

Practice these PracHub questions

These prompts let you practice the full architecture rather than isolated terminology. Say your assumptions aloud, draw the trust boundaries, and finish each answer with evaluation and failure recovery.

PracHub question	Practice focus	Why it helps
Design a RAG system with evaluation	Retrieval, grounding, citations, and per-stage metrics	Connects RAG architecture choices to measurable quality.
Design a Scalable and Safe Agentic System	Durable execution, permissions, budgets, and auditability	Tests how you bound autonomous behavior and recover from failures.
Design an Agent Harness and Evaluation System	Task sets, traces, graders, repeated trials, and release gates	Builds a rigorous answer for nondeterministic agent quality.
Design an enterprise RAG agent system	RAG versus agents, memory, tool safety, and concurrency	Combines the four themes in one enterprise design.

A seven-day preparation plan

Day	Focus	What to produce
Day 1	Problem framing	A one-page template for goals, risks, constraints, and success metrics.
Day 2	RAG	An ingestion and serving diagram with component-level metrics.
Day 3	Agents	A state machine with tools, approvals, retries, budgets, and termination.
Day 4	Evals	A versioned task set, grader plan, slices, and release threshold.
Day 5	Guardrails	A threat model for injection, data leakage, tool misuse, and recovery.
Day 6	Mock design	A 40-minute enterprise support-agent answer plus follow-ups.
Day 7	Review	A shorter second attempt that fixes unclear trade-offs and missing metrics.

Frequently asked questions

What is asked in a generative AI system design interview?

Expect an open-ended product or platform prompt involving retrieval, models, tools, evaluation, serving, or safety. The interviewer usually cares more about requirements, trade-offs, failure handling, and measurement than a specific vendor stack.

How is this different from a traditional ML system design interview?

Traditional ML interviews often emphasize data pipelines, training, features, deployment, and model monitoring. GenAI interviews add retrieved context, prompt and tool orchestration, nondeterministic trajectories, model-based evaluation, prompt injection, and controls for model-initiated actions.

Do I need a vector database in every answer?

No. Use one when semantic retrieval at the required scale and latency makes sense. Keyword search, relational filters, a knowledge graph, long context, or a combination may be better depending on freshness, access control, query shape, and operational complexity.

How should I evaluate a RAG system?

Measure ingestion quality, retrieval relevance, citation correctness, groundedness, answerability, task success, latency, and cost. Include no-answer, stale-document, permission-boundary, multilingual, and adversarial cases instead of relying only on a clean benchmark.

How do I prevent unsafe agent tool use?

Keep authorization outside the model. Use narrow tools, typed schemas, identity-aware permission checks, least privilege, idempotency, spend and step limits, sandboxing, approvals for consequential actions, and trace-based monitoring.

How much coding is expected?

The main round is often architectural, but follow-ups may ask for API schemas, data models, pseudocode for an orchestrator, retry logic, an evaluation harness, or a retrieval function. Be ready to make interfaces and failure semantics concrete.

Final takeaway

A convincing generative AI design is not the one with the most agent boxes. It is the one that uses the minimum necessary model autonomy, grounds decisions in authorized evidence, measures behavior continuously, and contains failures before they become user harm.

Practice the four linked PracHub prompts until you can connect every architecture choice to a metric, control, or recovery path. That is the difference between describing an AI demo and designing a production system.

Sources and Further Reading

Research note: This guide was checked on August 24, 2026. Tooling and model capabilities change quickly, but the architecture, evaluation, and risk-management principles remain broadly applicable.