Choose Between Fine-Tuning and RAG for Client Chatbot

Quick Overview

This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Choose Between Fine-Tuning and RAG for Client Chatbot states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Choose Between Fine-Tuning and RAG for Client Chatbot

Company: Amazon

Role: Data Scientist

Category: Machine Learning

Difficulty: hard

Interview Round: Technical Screen

##### Scenario Case study: choosing between fine-tuning and RAG for a client chatbot and improving retrieval quality. ##### Question When building an LLM application for a client, how would you decide between fine-tuning and Retrieval-Augmented Generation? List and compare fine-tuning methods such as full tuning, instruction tuning, LoRA and embedding fine-tune. Explain LoRA’s mechanism and its inference-time advantages. If retrieved documents show low relevance, how would you improve retrieval quality? The embedding model is the bottleneck; how would you fine-tune it? What data and training procedure are required? How would you architect a chatbot capable of answering questions across multiple knowledge domains? ##### Hints Compare approaches on cost, data needs, latency; propose iterative retrieval+model tuning and evaluation.

Quick Answer: This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Choose Between Fine-Tuning and RAG for Client Chatbot states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Machine Learning/Amazon
Amazon logo
Amazon
Aug 4, 2025, 10:55 AM
hardData ScientistTechnical ScreenMachine Learning
9
0

Choose Between Fine-Tuning and RAG for Client Chatbot

Scenario

You are building a client-facing chatbot that must answer questions grounded in the client's proprietary documents. You must choose how to imbue the system with domain knowledge and ensure high-quality retrieval.

Tasks

  1. Decide between fine-tuning a language model vs. Retrieval-Augmented Generation (RAG). Compare on cost, data needs, latency, maintainability, and risk.
  2. List and compare fine-tuning approaches: full model fine-tuning, instruction tuning (SFT), LoRA/QLoRA adapters, and embedding-model fine-tuning. Explain when each is appropriate.
  3. Explain LoRA's mechanism mathematically and the advantages it offers at inference time.
  4. If retrieved documents show low relevance, propose concrete steps to improve retrieval quality.
  5. The embedding model is the bottleneck. Describe how you would fine-tune it: data requirements, training objective, negatives, and evaluation.
  6. Propose a high-level architecture for a chatbot that must answer across multiple knowledge domains (e.g., product docs, policies, tickets), including routing and evaluation.

Clarifying Questions to Ask Guidance

  • Clarify the task, data shape, labels, constraints, and evaluation metric.
  • State assumptions behind the math or modeling technique you choose.
  • Connect theory to practical training, debugging, and deployment implications.

What a Strong Answer Covers Guidance

  • Correct definitions and formulas where the prompt requires them.
  • A practical explanation of how the method behaves on real data.
  • Trade-offs, failure modes, diagnostics, and mitigation strategies.
  • Evaluation choices that match the product or modeling objective.

Follow-up Questions Guidance

  • How would noisy labels, class imbalance, or distribution shift affect the answer?
  • What would you monitor after deployment?
  • Which baseline would you compare against first?
Loading comments...

Browse More Questions

More Machine Learning•More Amazon•More Data Scientist•Amazon Data Scientist•Amazon Machine Learning•Data Scientist Machine Learning