How to Build an AI Assistant for Your Internal Knowledge Base

"Code Llama is a state-of-the-art LLM for coding, capable of generating code and natural language about code from both code and natural language prompts." — Meta AI Blog, August 24, 2023

The most common first AI project we help businesses build right now is some version of this: "We have a lot of internal documents — wikis, SOPs, product specs, past proposals — and we want employees to be able to ask questions and get useful answers."

It's a good first project. The use case is genuinely valuable, the scope is containable, and success is relatively easy to measure (people either find it useful or they don't). It's also a project where the technical choices matter a lot and where there are specific pitfalls that trip up teams who haven't done this before.

This post walks through the complete build: what it takes, what it costs, and where it commonly goes wrong.

Step 1: Define the Knowledge Scope

Before you touch any technology, you need to decide: what knowledge does this assistant cover?

This sounds obvious, but it's where many projects stumble. The temptation is to say "everything" — load every document the company has ever produced into the system. Resist this.

Start narrower:

• A specific department's documentation
• A single product's technical specs and FAQs
• Your employee onboarding materials
• Your client proposal library

A narrower scope means:

• Better retrieval quality (fewer irrelevant documents to confuse the search)
• Faster build time
• Clearer success criteria
• Easier to identify what's missing

Document your scope as a list of specific document types and sources. This becomes your data inventory.

Step 2: Audit and Prepare the Documents

This is always more work than anyone budgets for. Plan accordingly.

Common problems we find:

• Documents in formats that are hard to parse (scanned PDFs without OCR, proprietary CRM exports, embedded spreadsheets)
• Outdated documents that would give wrong answers if retrieved
• Duplicate content across multiple versions of the same document
• Inconsistent language and terminology across departments

For each document source, you need a plan for:

• How to extract clean text
• How to handle updates (when the source document changes, the index needs to update)
• What metadata to preserve (author, date, department, document type) — this metadata enables filtering later

Budget at least 20-30% of your project time for data preparation. We often see it run 50%.

Step 3: Build the Ingestion Pipeline

The ingestion pipeline is the automated process that takes your documents and loads them into the vector database. The components:

Build this as a proper pipeline, not a one-time script. Documents change. New documents are added. The ingestion process needs to be repeatable and reliable.

Step 4: Build the Retrieval and Response Layer

When a user asks a question:

1. Embed the question using the same embedding model
2. Query the vector database for the top-k most similar chunks (typically k=3 to 5)
3. Assemble a prompt that includes: system instructions, the retrieved chunks, and the user's question
4. Send to the language model (GPT-4, Claude, or a self-hosted model)
5. Return the response

The system prompt is important. It should instruct the model to:

• Answer based only on the provided context
• Say "I don't know" or "I couldn't find that information" when the retrieved context doesn't contain a relevant answer
• Not hallucinate information that isn't in the retrieved documents
• Cite the source document where possible

Source citations are particularly valuable for internal tools — users want to know which document an answer came from so they can verify and read further.

Step 5: Build the Interface

For internal tools, a simple chat interface is usually right. Options:

• Slack/Teams bot — meets users where they already work, zero adoption friction
• Simple web app — more flexibility for displaying sources, structured results
• API only — if this assistant will be called by other internal tools

We've found Slack bot integrations have the highest adoption rates for internal knowledge tools. The friction of opening a new app is real.

Step 6: Evaluate Before You Launch

Before internal users rely on the system, you need to know how well it actually works. Run a structured evaluation:

Create a test set of 30-50 questions you know the answers to, drawn from your document library. Run them through the system. Score the responses:

• Correct and based on the right source ✓
• Correct but missing the source reference ≈
• Partially correct ≈
• Incorrect or hallucinated ✗
• "I don't know" when it should know ✗

Aim for >80% correct before launching to users. If you're below that, the issues are usually in data quality or chunking strategy, not the model.

Step 7: Monitor and Improve

After launch, collect:

• User queries (with consent and disclosure)
• Feedback signals (thumbs up/down, or just monitoring which queries users repeat, suggesting they weren't satisfied)
• Which queries return no useful results

This data is how you improve the system over time: add missing documents, improve chunking for specific document types, refine the system prompt.

Realistic Cost Estimates

For a small-to-medium business with ~500-1,000 internal documents and ~50 active users:

• Infrastructure: $100-300/month (Pinecone starter + OpenAI API + hosting)
• Build time: 6-10 weeks for a solid production system (including data prep)
• Ongoing maintenance: 2-4 hours/month to monitor, update documents, refine

The ROI math is usually straightforward: if 50 employees each save 30 minutes per week using the system, that's 25 hours of productivity per week. Even at a conservative loaded cost of $50/hour, that's $1,250/week — against infrastructure costs of less than $100/week.

The build investment pays back quickly.

Want to implement AI in your business? Book a 30-minute architecture call at blok-ai.com.