Building a RAG System From Scratch: A Business Owner's Guide

"Llama 3 sets a new standard for open-source AI, delivering state-of-the-art performance across a broad suite of industry benchmarks with improved reasoning, code generation, and instruction following." — Meta AI Blog, April 18, 2024

RAG — Retrieval-Augmented Generation — has become the foundational technique for business AI. It's how you give a language model access to your specific knowledge. It's why your AI assistant can answer questions about your actual products, policies, or customer history instead of just hallucinating generic answers.

If you're a business owner or executive thinking about building a RAG system, this post is for you. Not a technical deep-dive — that's for your developers. This is the business-level understanding you need to make good decisions about scoping, budgeting, and evaluating the work.

What RAG Actually Does, in Plain Language

A language model trained on the internet knows a lot about the world but nothing about your business. RAG fixes this by:

1. Taking all your relevant business knowledge (documents, databases, wikis, specs)
2. Processing and indexing it so it's searchable by meaning, not just keywords
3. At the moment a user asks a question, finding the most relevant pieces of your knowledge
4. Feeding those pieces to the language model along with the question
5. Having the model answer based on what it found

The result: an AI that behaves like a knowledgeable expert in your domain, not a generic chatbot.

The Four Decisions You Need to Make

Decision 1: What Knowledge Does It Cover?

This is the most important decision and it should be narrow at first. Good starting scopes:

• All documentation for a single product or service
• Your employee handbook and HR policies
• Your sales playbook and competitive battle cards
• Your technical support troubleshooting guides

Bad starting scope: "everything we have." This leads to poor retrieval quality, longer build time, and unclear success criteria.

The knowledge scope drives everything else — what data you need, how long the build takes, and how you measure success.

Decision 2: Who Are the Users?

Internal users (employees) or external users (customers) have very different requirements:

Start internal if you're not sure. It's lower risk and gives you time to improve before customers see it.

Decision 3: What Are Your Data Privacy Requirements?

Your documents are going somewhere to be processed. Understand where:

• Standard API (OpenAI, Anthropic): Your data passes through their APIs. Both have enterprise data processing agreements that restrict training use. Check the terms.
• Azure OpenAI or Bedrock: OpenAI/Anthropic models hosted on Microsoft/Amazon infrastructure. Better for enterprises with existing cloud agreements.
• Self-hosted model: Your data never leaves your infrastructure. Required for genuinely sensitive data. More complex and expensive to build and run.

Decide this early — it affects architecture choices significantly.

Decision 4: What Does Success Look Like?

Be specific before you start:

• "Employees get accurate answers to 80% of policy questions without contacting HR"
• "Customer support handles 40% of common queries without human escalation"
• "Time to answer a product specification question drops from 15 minutes to under 1 minute"

Without a specific definition, you can't measure, you can't iterate, and you can't make the case for expanding the system.

What It Costs

For a typical SMB RAG system (internal knowledge base, 200-500 documents, 30-100 users):

What drives cost up:

• Multiple data sources requiring different extraction logic
• Poor data quality requiring significant cleaning
• Complex authentication and access control requirements
• High query volumes requiring infrastructure scaling
• External/customer-facing deployment with higher reliability requirements

What drives cost down:

• Well-structured existing documentation
• Single data source with clean text
• Internal users who tolerate some imperfection
• Existing cloud infrastructure to build on

How Long It Takes

Realistic timelines for a straightforward project:

• Week 1-2: Data audit, scoping, architecture decisions
• Week 3-4: Data ingestion pipeline build
• Week 5-6: Retrieval and response layer, initial testing
• Week 7-8: Interface build, user testing, iteration
• Week 9-10: Evaluation, refinement, soft launch

So roughly 8-12 weeks from kickoff to a production-quality internal tool. External-facing systems typically add another 4-6 weeks for more robust testing, escalation paths, and compliance review.

Projects that try to rush this, especially the data preparation phase, almost always have quality problems that require going back and doing it right.

The Questions to Ask Your Developer or AI Consultant

• How will you handle updates when our documents change?
• What's the process for adding new documents to the knowledge base?
• How will we know if the system is retrieving the wrong documents?

• How will we evaluate accuracy before launch?
• What's the process for improving retrieval quality over time?
• How does the system handle questions that aren't in its knowledge base?

• What monitoring will be in place?
• What does our monthly maintenance look like?
• What's the escalation process when the system gets something wrong?

• What are the per-query costs at our expected volume?
• What happens to costs as usage scales?
• What's included in the build price vs. what's extra?

The Red Flags

Things that should make you pause:

• "We'll connect it to all your data" without a specific data assessment plan
• No discussion of evaluation or quality testing before launch
• Build timeline under 4 weeks for anything non-trivial
• No plan for the system to handle questions it can't answer
• No mention of how the system handles wrong or outdated information in your documents

The Green Flags

Things that indicate a competent team:

• They ask more questions than they answer in the first call
• They propose starting with a narrow scope and expanding
• They insist on an evaluation process before launch
• They have a specific plan for data preparation
• They can show you a similar system they've built before

Building a RAG system right is genuinely valuable. Building it wrong is expensive and leaves your business with a tool nobody trusts. Ask the hard questions upfront.

Want to implement AI in your business? Book a 30-minute architecture call at blok-ai.com.