Choosing Between Prompting, Retrieval and Fine-Tuning to Specialize an LLM

Starting from a capable general-purpose generative AI model, how would you adapt it so that it does well on work in one specialized domain? ```hint Start from the failure Before naming a technique, look at what the base model actually gets wrong on real tasks from this domain, and ask what kind of gap each failure reveals. ``` ### Clarifying Questions - Which domain and task are we targeting, for example drafting contracts, summarizing clinical notes, or answering questions about one company's internal documentation? - Can we change the model's weights, or can we only call it through an API with whatever customization the provider offers? - What domain data do we have: a document corpus, worked input-output examples, or time from experts who can grade outputs? - How quickly does knowledge in this domain change, and how costly is a wrong answer? ### What a Strong Answer Covers - A diagnosis of the model's current gap, grounded in a domain evaluation set built before any adaptation - A choice among adaptation methods, justified by the type of gap, the available data, model access and cost - Sourcing, quality, licensing and privacy of domain data, including how expert judgment is captured - Evaluation of domain gains alongside regressions in general ability and safety - How the adapted model is served, refreshed and monitored once it ships ### Follow-up Questions - After fine-tuning on domain examples, the model writes in the right style but states wrong domain facts more often than before. What went wrong, and what would you change? - A stronger base model is released after you shipped your adapted version. How do you decide whether to move to it, and what in your setup makes that move cheap? - Domain experts often disagree with each other about what a correct answer is. How do you build an evaluation set you can trust?