LLM Application Engineering: RAG vs Fine-Tuning, Context Limits, State and Prompt Injection

A technical screen for an applied AI engineering role included a round of rapid-fire conceptual questions about building applications on top of large language models (LLMs). Answer each of the four questions below in two or three minutes, concretely enough that the interviewer can tell you have built or operated such a system rather than read about one. ### Clarifying Questions - Should the answers assume a particular kind of application, such as a customer-support assistant or an agent that calls tools, or may I pick my own example? - Are we calling a hosted model through an API, or serving our own models? This changes whether fine-tuning and the context window size are under our control. ### Part 1 — RAG and fine-tuning What is retrieval-augmented generation (RAG), and how is it related to fine-tuning? When would you choose one, the other, or both? ```hint Ask where the new knowledge lives Compare where the added knowledge or behavior ends up in each approach, and what it takes to update it when it changes. ``` #### What This Part Should Cover - How a RAG pipeline works, from indexing documents to generating an answer - What fine-tuning changes in the model, and what it is and is not good at - Criteria for choosing between them, and ways to combine them - How you would evaluate either one ### Part 2 — Context limit exceeded Your LLM application starts failing because some requests exceed the model's context window. How do you handle it? ```hint Measure before you cut Start by working out which parts of the prompt consume the tokens, and which of those parts the model actually needs for this particular request. ``` #### What This Part Should Cover - Detecting the problem before the API call fails, and budgeting tokens across the parts of the prompt - Strategies for reducing what goes into the context, and their costs - Handling inputs that are inherently larger than any context window - Graceful behavior toward the user when nothing else works ### Part 3 — Making a stateless LLM application stateful LLM calls are usually stateless. How would you make an LLM application stateful, for example so that it remembers a conversation, a user's preferences, or the progress of a multi-step task? ```hint Name the state first List the kinds of state the application needs to remember, and for each one decide where it lives between requests and how it gets back into the model's input. ``` #### What This Part Should Cover - Where state is stored and how each request is rebuilt from it - Short-term conversation memory versus long-term memory across sessions - Task state for multi-step or agent workflows, including resuming after failure - Concurrency, privacy and cost concerns that come with stored state ### Part 4 — Prompt injection in an AI agent system How would you handle prompt injection in an AI agent system, one in which the model reads content from outside sources and can call tools? ```hint Assume the injection gets through Suppose some injected text reaches the model and it follows it. Ask what the agent could then do with its tools and data, and what would limit the damage. ``` #### What This Part Should Cover - The threat model: where injected instructions can come from and what an attacker can achieve - Why instructions in the prompt alone cannot solve the problem - Layered defenses around the model, its tools and its data flows - Detection, testing and monitoring in production ### What a Strong Answer Covers - Concrete mechanisms and examples rather than buzzwords - Trade-offs for each technique in cost, latency, quality and maintenance - Awareness of failure modes and how they would show up in production - How you would measure whether each fix actually worked ### Follow-up Questions - Your RAG system confidently answers with outdated information. How do you find out which stage went wrong? - How would you check that summarizing older conversation turns has not hurt answer quality? - Two browser tabs send messages to the same conversation at the same moment. What happens to your stored state? - The agent can send email on the user's behalf. Which of your prompt-injection defenses still hold if the model does follow an injected instruction?