"ChatGPT reached approximately 100 million weekly active users as of late 2023, with enterprise adoption growing more than 100% year-over-year." — OpenAI, January 2024
The market for AI consulting has exploded. Every agency that was doing digital marketing six months ago now has an "AI practice." Every freelance developer who's called the OpenAI API twice has updated their LinkedIn to include "AI/ML specialist." And a lot of businesses are getting burned.
I'm writing this because I think the market needs better filtering. Not because we need less competition — good AI consulting work is genuinely hard and the demand is real — but because bad AI consulting work is setting back the businesses that experience it.
Here are the questions I'd ask before hiring any AI consultant or studio, including us.
Question 1: "Show me a production system you've built."
Not a demo. Not a prototype. Not a POC. A production system — something running in an actual business, used by real employees or customers, with monitoring, error handling, and ongoing operations.
The gap between demo and production is enormous. It's the difference between showing that something works in ideal conditions and making it reliable, maintainable, and actually useful in the messy reality of a real business.
A good consultant should be able to walk you through the specifics: what the system does, what went wrong during build, how they handled edge cases, what the monitoring looks like, and what the client measured to assess success.
Red flag: consultant says they have "extensive AI experience" but describes only projects that were POCs, demos, or "helping clients evaluate" options.
Question 2: "What's the most common reason AI projects fail, in your experience?"
This question reveals epistemics. A consultant who's actually done the work will have a concrete, nuanced answer: data quality, scope creep, unclear success criteria, lack of human oversight, failure to account for model reliability. They'll have specific stories.
A consultant who's been reading about AI rather than doing it will give you a generic answer about "change management" or "AI adoption challenges" — true-ish but vague, and not grounded in actual project experience.
Listen for specificity. Listen for humility about things that went wrong. Listen for what they learned.
Question 3: "How do you decide whether RAG or fine-tuning is the right approach?"
This is a technical question but the answer tells you a lot about whether the person thinks clearly about AI systems. Both RAG and fine-tuning are methods for customising AI for a specific use case, and choosing between them requires understanding the actual problem.
A good answer will ask clarifying questions before answering: What kind of knowledge does the system need access to? Does it change frequently? Do you need citations? Do you need specific output formatting?
A bad answer will advocate for one approach without asking about your specific situation, or will indicate they don't understand the distinction.
Question 4: "Walk me through your data assessment process."
AI systems run on data. A consultant who can't describe a clear process for evaluating your existing data, identifying what's needed, and planning how to get it into usable shape is going to hit walls mid-project.
The process should include: understanding what data sources exist, assessing data quality, planning ingestion and processing, designing update pipelines for when data changes. It should acknowledge that data prep often takes longer than the AI build itself.
Red flag: the consultant minimises the data conversation or treats it as a simple step.
Question 5: "How do you handle model reliability and hallucinations in production systems?"
Any AI consultant who treats hallucinations as a solvable problem that the right model will eliminate is not someone you should trust to build a production system. Hallucinations are an inherent property of current language models. The right answer involves: guardrails, output validation, human review for high-stakes decisions, monitoring for error patterns, and design choices that constrain what the AI is asked to do.
Listen for a systematic approach to reliability. If the answer is essentially "the model usually gets it right," walk away.
Question 6: "What would you not use AI for in a situation like mine?"
The willingness to say "AI is probably not the right tool here" or "this use case is lower priority than it sounds" is a marker of intellectual honesty. It's also, counterintuitively, a good signal of competence — consultants who understand AI well enough to know its limits are better at applying it where it works.
If every question gets answered with "yes, AI can do that," you're talking to a salesperson, not a practitioner.
Question 7: "What does your post-launch support look like?"
AI systems need ongoing care: monitoring, prompt refinement, knowledge base updates, model updates, evaluation of output quality over time. A consultant who hands over a system and disappears is leaving you with a maintenance problem.
Ask specifically: What monitoring will be in place? Who handles issues? What's the process for improving the system over time? Is there a support retainer, or are updates billed separately?
The Broader Due Diligence
Beyond these questions:
- Check references. Talk to previous clients. Ask them what went wrong, not just what went well.
- Look at portfolio work. Does it match the complexity and type of what you're trying to build?
- Assess cultural fit. AI implementation requires close collaboration. Do they communicate well? Do they ask good questions?
- Understand the team. In smaller consultancies, the person you meet in the sales process may not be the person who does the work. Find out.
The market will mature and the filtering will get easier over time. Until it does, these questions will help you separate the consultants who've actually shipped AI from the ones who've only dreamed about it.
Want to implement AI in your business? Book a 30-minute architecture call at blok-ai.com.