"Meta's Llama 4 Scout features a 10 million token context window and outperforms GPT-4o and Gemini 2.0 Flash on key benchmarks, setting a new bar for open-weight models." — Meta AI Blog, April 2025
Your RAG system launched successfully. Users were happy. Queries were getting answered. Then, gradually, around month two or three, something changed. Users stopped using it as often. You started hearing "the AI gave me wrong information." The trust that took weeks to build started eroding.
This is one of the most common post-launch patterns we see. Not a dramatic failure — a slow degradation that happens when the post-launch care and feeding that the system needs doesn't happen.
Here are the mistakes that cause it, and what to do about each.
Mistake 1: The Knowledge Base Went Stale
This is the most common one. RAG systems answer questions based on their indexed documents. When those documents are updated — new product versions, revised policies, changed pricing, updated procedures — the knowledge base needs to update too.
If the knowledge base update process is manual, it almost certainly hasn't been done consistently. Which means users are now getting answers based on outdated information.
The failure pattern: A user asks about the current return policy. The knowledge base has the policy from six months ago. The policy changed three months ago. The AI gives the old answer with complete confidence. The fix: Build automated update pipelines from day one, not retroactively. For documents stored in specific systems (Notion, Confluence, SharePoint, Google Drive), build webhook or scheduled sync processes that update the vector index when source documents change. For documents that live outside managed systems, create a clear process and designate someone responsible for triggering updates.As a diagnostic: right now, list every document in your knowledge base and identify which ones have changed since initial indexing. The answer is often sobering.
Mistake 2: Nobody Is Monitoring Retrieval Quality
The retrieval layer — the part that finds relevant documents before the model generates an answer — can silently degrade in quality without anyone noticing. Common causes: new types of queries that don't match the existing document structure, edge cases where the most-retrieved document is semantically similar but actually irrelevant, gaps where relevant information simply isn't in the index.
Users experience this as "the AI gave me a wrong answer" or "the AI made something up." But it wasn't hallucination from the model — it was retrieval returning the wrong context, and the model trying to make the best of it.
The fix: Log retrieved documents for every query (you're probably already logging queries). Periodically sample these logs — 20-30 queries per week — and check: is what was retrieved actually relevant to the question? If retrieval quality is declining, the usual culprit is a gap between query language and document language that emerged as users explored more of the system's capabilities.Fix options: query reformulation (use the LLM to rewrite queries before embedding), adding missing documents that cover the new query types, adjusting chunking strategy for specific document types, or adding keyword search as a hybrid alongside vector search.
Mistake 3: The System Prompt Hasn't Been Updated
The system prompt — the instructions to the model about how to behave — was written when the system launched. User behavior and expectations have evolved. New use cases emerged that the original prompt didn't account for.
Common symptoms: the model starts answering off-topic questions it was supposed to decline, or conversely, refuses reasonable questions because they superficially resemble something the prompt told it to avoid. The "personality" of the AI assistant drifts from what users expect.
The fix: Review the system prompt quarterly. Does it still accurately describe the scope of the system? Does it handle the edge cases that have emerged in real usage? Are there new failure modes that warrant explicit instructions?Treat the system prompt as maintained code. Version control it. Test changes against your evaluation set before deploying them. Document why specific instructions are in the prompt.
Mistake 4: The Evaluation Set Wasn't Updated
If you built an evaluation set at launch (which you should have), it represented the kinds of queries you anticipated at that time. Real users have likely asked things you didn't anticipate. The new failure modes aren't captured in your evaluation set, which means your quality metrics look fine even as real-world quality has declined.
The fix: Expand your evaluation set regularly. Every quarter, review actual production queries and add the most representative new cases — including cases where the system failed — to your evaluation set. Your evaluation set should grow over time to cover the evolving distribution of real queries.Mistake 5: User Feedback Isn't Being Acted On
Most RAG systems have some feedback mechanism — thumbs up/down, star rating, explicit feedback form. Most of that feedback data is sitting in a database that nobody is reviewing.
When users mark a response as unhelpful, they're giving you specific, actionable information about where the system is failing. Ignoring it means missing your best signal for improvement.
The fix: Designate someone responsible for reviewing user feedback weekly. Create a simple triage: "content gap" (the information simply isn't in the knowledge base), "retrieval failure" (the information is there but wasn't retrieved), "generation failure" (retrieval was right but the model response was poor), or "expectation mismatch" (the system worked correctly but the user expected something different). Each category has a different fix.Mistake 6: Edge Case Handling Isn't Graceful
When the system can't answer — because the information isn't in the knowledge base, or because the query is out of scope — what does it do? If the answer is "it tries anyway and often gets it wrong," you have a trust problem.
Early in the system's life, edge cases are rare and don't define the experience. By month three, users have explored more of the system's capabilities and are hitting edge cases more frequently.
The fix: Build explicit "I don't know" detection. Test: what does the system do when asked a question that definitely isn't in the knowledge base? Ideally: acknowledges it doesn't have the information, explains what it does cover, and optionally suggests how the user might find the answer elsewhere. A system that admits its limits is trusted more than one that always tries to answer.The Monthly Maintenance Checklist
To keep a RAG system in good shape:
- Weekly: Review user feedback (15 minutes)
- Weekly: Sample 20 query logs and check retrieval quality (30 minutes)
- Monthly: Check for stale documents in the knowledge base
- Quarterly: Review and update system prompt
- Quarterly: Expand evaluation set with new cases, run evaluation
Total monthly time investment: 3-5 hours. The cost of not doing it: a slowly degrading system that users stop trusting.
RAG systems are not set-and-forget. The ones that stay valuable are the ones that get tended to.
Want to implement AI in your business? Book a 30-minute architecture call at blok-ai.com.