"Anthropic has raised $4 billion from Amazon, bringing total investment to over $7 billion and valuing the AI safety company at over $15 billion." — Reuters, October 2024
The AI consulting market has never been more crowded. Every digital agency, every solo developer, every strategy consultancy has rebranded to include "AI." In a market this noisy, separating capable from pretending is genuinely difficult.
This post is the practical filter I'd use if I were buying AI consulting services rather than providing them.
The Red Flags
They Lead With Technology, Not Problems
A consultant who spends your first call pitching specific technologies — "we use LangChain, we use Pinecone, we specialize in GPT-4" — before understanding your problems is doing it backwards. Technology is a consequence of problem definition, not a starting point.
The right consultant asks: what's the problem you're trying to solve? Who has it? How often? What would "solved" look like? What have you tried? Then, and only then, does technology enter the conversation.
Their Portfolio Is All Demos and POCs
You find their website or LinkedIn and see impressive demos — a chatbot that does something clever, a RAG system that knows about a specific topic, a well-produced video of an AI in action. But when you ask about production deployments, things get vague.
Building demos is a fundamentally different skill from building production systems. Production systems need error handling, monitoring, update pipelines, user feedback loops, graceful degradation, and reliability under load. A consultant who's only ever built demos doesn't know what they don't know.
Push for specifics: "Can you show me something that's been in production for more than six months? What were the most significant production issues you encountered?"
They Promise Specific Results They Can't Guarantee
"We'll reduce your support tickets by 40%." "We'll save your team 20 hours a week." "ROI within 3 months."
AI implementation results depend heavily on your specific data, your specific workflows, and your specific organisation's willingness to change. A consultant making specific quantitative promises before doing any discovery is either wildly optimistic or telling you what you want to hear.
Good consultants give ranges based on past experience, acknowledge dependencies, and are clear about what they can guarantee (they'll build a working system) versus what they can only influence (business outcomes).
They Don't Mention Failure
The best practitioners have stories of things going wrong. Data that turned out to be worse than expected. A model that performed well in testing and poorly in production. A project that needed significant rework mid-build. A use case that turned out not to be a good fit for AI.
A consultant who never mentions failures, only successes, hasn't done enough real work — or is curating their narrative too carefully for you to trust them.
They Can't Explain Their Evaluation Process
Ask: "How do you evaluate whether the AI system is actually working?" If the answer is vague — "we run tests," "we check with stakeholders," "we iterate based on feedback" — that's a red flag.
Good evaluation for AI systems requires: a defined test set with known correct answers, quantitative quality metrics, systematic coverage of edge cases, and a process for ongoing measurement after launch. If they can't describe this specifically, they're probably not doing it.
Their Team Is Thin
Some AI "studios" are one or two people who've subcontracted everything. That's not inherently bad, but you need to understand who's actually building your system. Ask for the team profiles of the people who will do the work, not just the partners who run the sales process.
The Green Flags
They Ask Hard Questions Early
A good consultant makes you work a little in the first conversation. They ask about your data situation, your existing systems, your internal technical capacity, your risk tolerance, your definition of success. They're not trying to qualify you out — they're trying to understand your situation well enough to give you an honest answer.
This is uncomfortable but valuable. The consultants who ask hard questions early are the ones who surface the real challenges before they become expensive mid-project surprises.
They're Honest About What Won't Work
The willingness to say "that's not a good use case for AI, and here's why" is a strong signal of genuine expertise. Practitioners who understand the technology well enough know its limits — not just its capabilities.
If every question you ask gets answered with "yes, we can do that," you're probably talking to a salesperson. If you occasionally hear "actually, for that specific case, a simpler approach might be more reliable," you're talking to someone who's thought about this carefully.
They Have Specific Answers to Technical Questions
You don't need to be technical yourself to ask technical questions and listen for specificity. Ask about their chunking strategy for RAG systems — do they give a specific answer, or a vague one? Ask about how they handle model reliability — do they describe specific guardrail approaches, or generalities?
Specific, confident answers to technical questions suggest someone who's actually done the work. Vague, general answers suggest someone who's read about it.
They Have References Who Picked Up the Phone
Not just references who can be provided — references who actually talk to you in meaningful detail about the project. Ask what went wrong, not just what went right. A reference who says "it was smooth the whole way" is probably not being fully candid. A reference who says "there was a challenging period around week 6 where the data turned out to be messier than expected, but they handled it well" is giving you real information.
They Have Domain Context Relevant to Your Business
An AI consultant who's built healthcare AI doesn't automatically understand financial services AI. Domain context matters — not because the technical skills transfer differently, but because understanding what failure looks like and what accuracy requirements actually mean is domain-specific.
Especially relevant for Japan-market work: working with a consultant who understands Japanese business culture, data handling expectations, and can work in Japanese when needed is a meaningful differentiator. Not because consultants without Japan experience can't do good work, but because the cultural and linguistic context adds real value.
They Propose Starting Small
A consultant who proposes beginning with a well-scoped Phase 1 rather than jumping directly to maximum scope is showing good judgment. The best way to evaluate an AI implementation partner is to see them in action on a contained project. A good consultant knows this and proposes it themselves.
The Process
Finding the right partner takes more effort than it should in a mature market. For now, here's the practical approach:
- Get 3-4 referrals from people who've actually used AI consultants, not just heard about them
- Run first calls with all of them before any proposal stage
- Ask the hard questions from this post
- Request specific references and actually call them
- If possible, commission a small scoping project before a full engagement
The right AI consulting partner makes your project significantly more likely to ship, to work, and to create real business value. That's worth taking the time to find.
Want to implement AI in your business? Book a 30-minute architecture call at blok-ai.com.