When an LLM Feature Should Use Prompting or Fine-Tuning Instead of Retrieval

You are interviewing for a product manager role at Google, and the conversation turns to products built on large language models. Retrieval-augmented generation (RAG) looks up relevant documents when a request arrives and places them in the model's context before it answers. The interviewer wants to know when you would choose not to use it: describe the situations in which prompting alone, or fine-tuning a model, is the better choice for a feature, and explain how you would reach that decision. ```hint Start from the failure Before picking a technique, pin down what the model gets wrong today without any help, and ask which of the three approaches actually supplies the missing piece. ``` ### Clarifying Questions - Is the feature a high-volume, user-facing one where latency and cost per request are tight, or an internal tool where they matter less? - Does the knowledge the feature relies on change every day, or is it a small, stable body of material such as a single policy or style guide? - Can the team train and maintain its own tuned model, or is it limited to calling a hosted model with prompts? - Must answers cite their sources or respect each user's access rights to the underlying documents? ### What a Strong Answer Covers - A precise account of what each technique changes: the information the model sees on each request versus the behavior learned in its weights - Concrete situations where RAG is the wrong tool, each tied to a reason such as the size and freshness of the knowledge, the type of task, latency and cost, the quality of the corpus, or deployment limits - When prompting alone is enough, and what evidence justifies the extra cost and upkeep of fine-tuning - Product trade-offs: time to ship, serving cost, maintenance as base models change, data governance, and the ability to cite or delete information - How the choice is validated with an evaluation set, including cases where the approaches are combined ### Follow-up Questions - A help assistant must answer from one short refund policy that changes a few times a year. Which approach would you pick, and what would make you switch? - You fine-tuned a model on customer documents, and a customer now asks for their data to be deleted. What does that do to your choice of approach? - Which evaluation results would convince you that better prompting has reached its limit and fine-tuning is worth the cost?