"DALL·E 3 is built natively on ChatGPT, allowing users to use ChatGPT as a brainstorming partner and prompt refiner — raising the bar for text-to-image generation." — OpenAI Blog, September 2023
Every few weeks, a client comes to us with some version of this question: "Should we fine-tune a model or use RAG?"
It's a good question, and the answer is almost always "RAG — at least to start." But the reasoning matters, so let me walk through it properly.
Defining the Terms
RAG (Retrieval-Augmented Generation): Instead of training the model on your data, you keep your data in an external knowledge store (typically a vector database). At query time, you retrieve the most relevant pieces of your data and include them in the prompt context before the model generates a response. The model's weights never change. Fine-tuning: You take a pre-trained base model and continue training it on examples from your specific domain or use case. This modifies the model's weights — it changes how the model "thinks" based on your training data. After fine-tuning, the knowledge is baked into the model itself.Both approaches can make a language model more useful for your specific context. They do it in completely different ways, and they're good at completely different things.
What RAG Is Good At
Factual, document-based question answering. If a user is asking "what does our return policy say?" or "what were Q3 revenue numbers from the board report?", RAG is the right approach. The answer exists in a specific document. You want to retrieve that document and have the model synthesise an answer from it. Knowledge that changes. Your pricing changes. Your policies update. New products launch. With RAG, you update the knowledge base and the AI immediately has access to the new information. With a fine-tuned model, you'd need to retrain every time the knowledge changes — expensive and slow. Large and diverse knowledge sets. You can put thousands of documents into a RAG system. A fine-tuned model would struggle to reliably recall information from that many distinct sources, and the training data volume required would be enormous. Citability. RAG systems naturally know which source documents were used to generate a response. This is important for business contexts where users need to verify information or understand where an answer came from. Lower cost and faster iteration. RAG requires no training compute. You can build, test, and iterate in days. Fine-tuning on GPT-3.5 takes time, money, and experimentation to get right.What Fine-Tuning Is Good At
Style and tone. If you want the model to consistently write in your brand voice, format outputs in a specific way, or match the communication style of your company, fine-tuning is much more effective than prompt engineering. You train the model on examples of the style you want. Format adherence. If you need the model to always output structured JSON in a specific schema, or always produce reports with particular sections in a particular order, fine-tuning on examples of correct outputs improves reliability significantly. Domain-specific reasoning patterns. If your domain has patterns of reasoning that differ from general text — legal reasoning, medical assessment frameworks, specific engineering calculation approaches — fine-tuning on domain examples can embed those patterns into the model's behaviour. Capability uplift in specific tasks. If there's a specific narrow task (e.g., classifying customer support tickets into 15 categories) and you have thousands of labelled examples, fine-tuning can achieve higher accuracy than prompting a general model for that specific task. Smaller model performance. If you need to run a smaller, faster, cheaper model (important for cost at scale), fine-tuning can bring a smaller model up to a quality level that approaches a larger general model on your specific task.The Common Mistake
Most businesses come to us thinking they need fine-tuning when they actually need RAG. The reasoning usually goes: "We have proprietary data and we want the AI to know it." That's RAG, not fine-tuning.
Fine-tuning doesn't reliably give a model access to factual information from your training data. Language models are bad at memorising and retrieving specific facts — they're trained to generate plausible text, not to be databases. If you fine-tune on your product documentation, the model might learn to write in your product's style and vocabulary, but it will hallucinate specific facts just as readily as the base model would.
If you want the AI to know your specific facts, put those facts in a RAG knowledge base.
The Hybrid Approach
The most capable systems often use both:
- RAG for knowledge access — giving the model access to specific, up-to-date, citable information
- Fine-tuning for behaviour — teaching the model how to reason, structure outputs, and communicate in your context
For example: a financial services firm might fine-tune for their specific report format and analytical framework, while using RAG to provide access to current market data and client-specific information.
But start with RAG. Get it working. Then consider fine-tuning for specific behavioural improvements once you understand exactly what the base RAG system is getting wrong.
Decision Framework
Ask yourself:
- Do I need the AI to know specific facts from my documents? → RAG
- Does my knowledge change frequently? → RAG
- Do I need citations and source traceability? → RAG
- Do I need specific output formatting or tone? → Fine-tuning
- Do I have hundreds or thousands of labelled examples of correct behaviour? → Fine-tuning
- Am I trying to improve performance on a narrow, specific task? → Fine-tuning
- I'm not sure / this is my first AI project → RAG
The instinct to fine-tune is understandable — it sounds like "training AI on your data," which sounds powerful. And it is powerful, for the right use cases. But for 80% of business AI applications, RAG is the right first move.
Want to implement AI in your business? Book a 30-minute architecture call at blok-ai.com.