Choosing Between Retrieval-Augmented Generation and Fine-Tuning for an LLM Product

Explain when you would ground a large language model with retrieval-augmented generation (RAG) and when you would fine-tune it instead, and why. ```hint Ask what the model is missing Picture a concrete assistant that is falling short, and ask what would happen under each approach when the information it relies on changes next month. ``` ### Clarifying Questions - Should the answer assume a particular use case, such as an assistant that answers questions from frequently updated help or policy content, or stay general? - Must answers cite their sources, and do different users have permission to see different documents? - Are there limits on latency, serving cost, or whether customer data may be used to train models? - Is the choice strictly one or the other, or can the two be combined? ### What a Strong Answer Covers - A correct explanation of what each technique changes: the context the model reads at answer time versus the model's weights. - Decision criteria tied to the problem: how often the knowledge changes, citations and access control, consistency of behavior and format, available training data, latency and cost. - When to combine the two, and cheaper steps to try first, such as better prompting. - How to evaluate the choice, and the failure modes and maintenance costs of each approach. - A concrete product example that links the technical choice to user impact. ### Follow-up Questions - Your RAG assistant gives wrong answers even though the right document is in the index. How do you tell whether retrieval or generation is at fault? - Legal rules that customer data may not be used for training. What does that change in your approach? - How would you decide whether a fine-tuned smaller model can replace a large model that relies on long prompts, to cut cost?