"Two years after its launch, ChatGPT now has over 200 million weekly active users — doubling its user base in the past year as enterprise adoption accelerates." — OpenAI, November 2024
We've been shipping AI systems to production for over a year now. Not POCs, not demos — systems that run in actual businesses, handle real workloads, and get used every day. Some by dozens of users, some by hundreds.
Here's what we've learned about what actually happens when AI systems meet the real world at scale.
Lesson 1: The Evaluation Gap Is Real, and It's Expensive
Every system we've built has had some version of the same experience: it works beautifully in testing. The evaluation set shows 85%+ accuracy. The demo impresses everyone. Then production usage starts and the error rate is nothing like what testing suggested.
Why? Because test sets, however carefully constructed, don't capture the full distribution of real-world inputs. Users ask questions in ways you didn't anticipate. They use terminology from their domain that your test set didn't cover. They push the system in directions the testing process didn't consider.
What we do now: We run evaluation in three phases. First, a curated test set built before development to validate the architecture. Second, a "realistic adversarial" test set designed to break the system — edge cases, ambiguous inputs, questions the system shouldn't answer. Third, a production shadow mode where we log real queries and review them daily for the first two to four weeks before going fully live. The shadow mode catches what the other two phases miss.Lesson 2: Retrieval Quality Is the Dominant Variable in RAG Systems
In every RAG system we've built, the model choice and prompt engineering matter less than retrieval quality. A mediocre model with excellent retrieval beats an excellent model with mediocre retrieval on almost every metric.
And retrieval quality is harder to get right than it sounds. The issues we encounter most often:
Chunking strategy mismatches. Fixed-size chunking often splits semantic units in the wrong places. A policy document where each section answers a specific question needs section-aware chunking, not 500-token windows. Metadata filtering underused. We often build vector stores with good metadata (document type, date, department, topic) and then fail to use that metadata to filter retrieval. Filtering significantly improves precision. Query reformulation neglected. User queries in natural language often don't match well with how documents are written. Adding a query reformulation step — using a language model to rewrite the user's query in a way that better matches document language — can improve retrieval by 20-30%. Missing content gaps. The system confidently retrieves the most relevant thing in the index, even when the answer simply isn't in the knowledge base. Detecting "I don't know" situations reliably is a separate problem that needs explicit handling.Lesson 3: Latency Is User Experience
Users accept 1-2 second response times. Above 3-4 seconds, completion and engagement drop measurably. Above 5 seconds, users abandon and lose trust in the system.
This matters because many AI systems, especially ones routing through large models with extended context, have latency that can creep toward 5-10 seconds for complex queries.
What we do: Streaming responses (so users see text appearing rather than waiting for a full response), careful model tier selection (using faster, lighter models where quality permits), and aggressive caching of common queries. For the cases where latency is genuinely unavoidable, set user expectations explicitly — "Analysing your documents (typically 15-30 seconds)" is much better than a blank spinner.Lesson 4: You Need Error Budgets, Not Zero-Defect Goals
Early in our deployments, we'd see an error in production and treat it as an emergency. This is wrong and exhausting.
AI systems have inherent error rates. A RAG system with 85% accuracy means 15% of responses have some quality issue. The question is: what quality issues are acceptable, what's the appropriate response to each type, and what's the threshold that triggers an alert vs. a planned improvement sprint?
What we do: Define error classes before launch. Critical errors (wrong factual information, system fails to escalate when it should, security/privacy issue) get immediate response. Quality errors (answer is technically correct but poorly formatted, or the system misidentifies intent but recovers) get logged and addressed in the next improvement sprint. We track error rates over time and alert if they trend upward. This is so much more sustainable than treating every imperfect output as a crisis.Lesson 5: Prompt Engineering Is Maintenance, Not a One-Time Task
The system prompt — the instructions that tell the model how to behave — is not something you write once and forget. It needs maintenance like any code.
We've seen system prompts degrade in quality as the knowledge base changes, as model updates shift model behaviour slightly, and as edge cases emerge that the original prompt didn't account for. The prompt that worked perfectly in November can start producing borderline outputs in February if nothing else changed.
What we do: Version control all prompts (yes, even prompts). Run regression tests on prompts after any change. Review prompt performance monthly against the evaluation set. Treat prompt updates as code releases — reviewed, tested, deployed.Lesson 6: The Ops Layer Is Not Optional
Every production AI system we've built has the same non-negotiable operations requirements:
- Logging: Every query, every retrieved document, every model response. Essential for debugging and improvement.
- Cost monitoring: Token usage and API costs tracked per query, per user, per day. AI API costs can spike unexpectedly — you need to know before it shows up on the bill.
- Quality monitoring: Track output quality metrics over time. Set alerts if quality degrades.
- Update pipeline: When knowledge base documents change, what's the process for updating the index? Who triggers it? How do you verify the update worked?
The systems that degrade fastest are the ones where nobody built the operations layer. The knowledge base gets stale, nobody notices, and users stop trusting the system.
The Pattern That Works
The AI systems that are still running well 12 months after launch share the same traits: good evaluation processes that catch real-world failures, careful attention to retrieval quality, latency optimisation as a first-class concern, defined error budgets with appropriate responses, maintained prompts, and a real operations layer.
None of these are glamorous. None of them show up in demos. All of them matter enormously in production.
Want to implement AI in your business? Book a 30-minute architecture call at blok-ai.com.