Teams building on large language models (LLMs) keep arriving at the same fork: our model does not know our company’s information, so do we use retrieval-augmented generation (RAG) or fine-tune the model? Here is the position of this essay, stated early: for most business problems about knowledge, start with RAG. Reach for fine-tuning when your problem is about behaviour. The two solve different things, and confusing them is the expensive mistake.
What each one actually does
An LLM has two sources of information when it answers: what it absorbed during training, stored in its weights, and what you put in the prompt right now.
RAG works on the second source. When a user asks a question, your system first searches a collection of your documents, usually by converting text into numeric vectors (embeddings) and finding passages whose meaning is close to the question. It then pastes the best few passages into the prompt and asks the model to answer using them. The model’s weights are untouched.
Fine-tuning works on the first. You continue training the model on examples, adjusting its weights, so it responds in a particular style, format or specialised task. The usual input is many pairs of “this input, this ideal output”.
The case for RAG when facts are the problem
Take a support assistant for a bank’s retail loan products. Interest rates change, a policy document is revised every quarter, and an answer given in March should not quote February’s rate. With RAG you update a document in the index and the next answer reflects it, with no retraining. With fine-tuning, facts are baked into weights; updating them means preparing data and training again, and you cannot easily point to where a given fact lives or remove it.
RAG also helps with trust. Since the answer is built from retrieved passages, you can show the source document and page beside it. A user or an auditor can check. A fine-tuned model that “just knows” something gives you nothing to verify. For regulated settings in banking, insurance or healthcare, this traceability often decides the matter.
Access control is the third point, and it is easy to overlook. If different staff may see different documents, RAG can filter the retrieved passages by the user’s permissions before the model ever sees them. Once information is inside the model’s weights, you cannot hide it from some users and reveal it to others.
Fine-tuning for facts is also unreliable. Models trained on new documents may still answer wrongly with confidence, and the effect is hard to test. Research and practitioner reports have generally found that supplying knowledge in context is a more dependable route than trying to inject it through training, though results vary by model and task.
The case for fine-tuning when behaviour is the problem
Now imagine a different need: a model that reads messy customer emails and always returns a particular structured form, in your company’s labelling scheme, with your tone. The model does not lack facts here; it lacks a habit. Long instructions and examples in the prompt can do part of the job, but they cost tokens on every call and are sometimes ignored.
Fine-tuning suits this: consistent format, a specific classification scheme, a house writing style, domain-specific jargon, or making a smaller and cheaper model perform one narrow task nearly as well as a big one. Because the behaviour lives in the weights, prompts get shorter, which can cut latency and cost at high volume.
The price is real. You need good labelled examples (often hundreds to thousands, depending on the task), an evaluation set to prove it helped, and a plan to redo the work when the base model is replaced. Options like parameter-efficient methods (LoRA and similar) reduce compute cost, but the data and evaluation effort remain the hard part.
A quick way to tell which you need
| If the complaint is… | It is probably… | Lean towards |
|---|---|---|
| “It does not know our policy” | Missing knowledge | RAG |
| “The answer is out of date” | Changing knowledge | RAG |
| “We need to cite the source” | Traceability | RAG |
| “It will not follow our output format” | Behaviour | Better prompt first, then fine-tuning |
| “Too slow and costly at our volume” | Model size and prompt length | Fine-tune a smaller model |
Where RAG goes wrong
RAG is not magic, and teams who treat it as plug-and-play are disappointed. Most failures are retrieval failures, not model failures. If the right passage is never retrieved, the model either says it does not know or, worse, produces a plausible guess. Problems to watch for include documents chopped into chunks that lose their context (a table row without its heading), scanned PDFs with poor text extraction, questions that need information from several documents, and vocabulary mismatch between how users ask and how documents are written. Combining vector search with ordinary keyword search, and re-ranking the results, is a widely used improvement.
Evaluation matters more than choice of tools. Build a small set of real questions with known correct answers, and measure two things separately: did retrieval bring back the right passage, and did the model answer faithfully from it? Without that, you will argue from impressions.
They are not rivals
It is quite common to use both: a lightly fine-tuned model that reliably produces your format and tone, fed by RAG for current facts. But sequence matters. Begin with a good base model, careful prompts and RAG. Measure. Fine-tune only when you can name a specific behavioural gap the measurements reveal. Teams that start by fine-tuning on their documents often spend weeks to rediscover that their real need was a search problem.
Leave a Reply