Walk Through an Agentic RAG Assistant for On-Call Tickets and Chat Questions

After a short coding problem, the interviewer in this senior loop spends most of the round on an AI system listed on your resume. You are asked to draw its architecture and explain it end to end, and the conversation keeps returning to *why*: how the prompts were engineered, why each component was chosen, which technologies were picked, and what reasoning shaped the overall architecture. For practice, assume the project is an assistant that supports an engineering team's on-call rotation. It is built on retrieval-augmented generation (RAG) plus tool-using agents, and it does two jobs: 1. It handles lower-priority tickets that land in the team's queue. 2. It answers questions that stakeholders post to the team in a chat workspace such as Slack. If you have built a comparable system yourself, present your own instead; the interviewer is testing whether you can explain and defend decisions you actually made, not a textbook diagram. ### Constraints and Clarifications - Expect to draw the system and talk through it; code is not required. - The ticketing system, knowledge sources, model provider and traffic volume are not specified. State your assumptions and keep them consistent. - What it means for the assistant to "handle" a ticket is part of the design: decide what it may do on its own and what needs a human. ### Clarifying Questions - Which tickets are eligible for automatic handling, and what does handling mean here: drafting a reply, resolving and closing, or taking an action on a service? - What knowledge should the assistant draw on: runbooks, previously resolved tickets, design documents, service ownership data, dashboards? - Who asks questions in chat, and what do they usually want (status of an issue, how to use a service, who owns something)? - How quickly must a chat answer arrive compared with a ticket update? - Are there documents or tickets that some stakeholders must not see? ### Part 1 — Draw the system end to end Draw the architecture and walk through both flows: a new low-priority ticket arriving, and a stakeholder question posted in chat. Show how knowledge gets into the system, how a request is routed, what the agent can call, and how a result gets back to the ticket or the chat thread. ```hint Two entry points, one core Trace each flow separately from its trigger to its final output, then point out which components both flows share and which exist for only one of them. ``` #### What This Part Should Cover - Knowledge ingestion and indexing, kept separate from the request path - Routing and orchestration for the ticket flow and the chat flow - The agent's tools and the boundary between read-only and state-changing actions - How answers are delivered back, and where a human stays in the loop ### Part 2 — Defend the design choices The interviewer now probes the reasoning. Explain how you engineered the prompts, why you chose each major component (retrieval approach, model, an agent framework or a hand-written orchestrator), which technologies you picked, and which alternatives you rejected and why. ```hint Tie each choice to a failure For every component, name the problem it was introduced to fix or the risk it reduces; a choice you cannot connect to a concrete problem will not survive follow-up questions. ``` #### What This Part Should Cover - Prompt structure: instructions, retrieved context, output format, refusal and escalation behavior - Retrieval design choices, and why RAG fits this knowledge better than fine-tuning - When an autonomous agent loop is justified versus a fixed pipeline - Concrete alternatives considered, with the trade-off that decided each one ### What a Strong Answer Covers - A clear, layered diagram that the candidate can narrate without losing the thread - Guardrails proportional to risk: scoped tools, approval for writes, escalation to the on-call engineer - A way to measure quality: offline evaluation on past tickets and online outcome signals - Operational concerns: tracing of retrieval and tool calls, cost and latency, stale knowledge - Honest ownership: which parts the candidate built and decided, as distinct from the team's work ### Follow-up Questions - How do you stop instructions hidden inside a ticket or a document from steering the agent into an unsafe action? - The assistant confidently closed a ticket that turned out to be the first sign of an outage. What would you change? - How would you evaluate a new prompt or model version before rolling it out? - How do you keep the index current when runbooks change, and how do you prevent answers built on outdated documents?