Agency16 June 20264 min read

How to evaluate an AI consulting partner: a CTO's checklist

Most AI consultancies sell decks or dependency. A few transfer real capability onto your team. Six questions that tell the difference before you sign.

The short answer

Score them on capability transfer, not the pitch. Ask whether they build or only advise, whether they work on your existing stack, whether your team owns the result after they leave, whether they augment your developers or replace them, and whether they can show production work, not demos. A good partner makes you independent.

tsukumo

How to evaluate an AI consulting partner: a CTO's checklist

Short version: most AI consultancies sell one of two things, a strategy deck or a dependency, and both leave your team where it started. A few actually transfer the capability onto your developers, so you own it when they leave. The way to tell them apart isn't the pitch. It's six questions: do they build or only advise, do they work on your stack or impose a rebuild, does your team own the result, do they augment your developers or replace them, can they show production rather than demos, and are they honest about when you don't need them.

One thing up front. We're a firm in this market, so I'm writing the checklist that vets people like us. That's a conflict, and I'd rather name it than bury it. So I built these questions to cut both ways: a few of them would knock us out on a bad week, not just our competitors. If they only ever flatter the author, they're marketing. Use them on us too.

Start with what you're actually buying#

Before you compare vendors, name the thing you want to walk away with. It usually isn't a roadmap, and it isn't seats. It's capability: your team able to run AI agents on your codebase, in production, after the engagement ends. Hold every partner against that outcome. A lot of spend in this market buys activity that looks like progress (workshops, audits, a platform login) without moving you closer to it.

1. Do they build, or only advise?#

The market splits here. Strategy-first firms are strong at defining what to do and weak at the part that actually fails: getting agents reliable on a real codebase. They hand you specs and the production gap stays yours. A build partner works in code, hits the same failure modes your team would, and solves them where they happen. For getting AI into production, building beats advising. Ask to watch them work on something real, not present about it.

2. Do they work on your stack, or impose a rebuild?#

Your team has an existing environment, standards, and a review culture that works. A partner who needs to replace all of that to add AI is solving their problem, not yours. The honest version meets your repo as it is, respects how your team already ships, and adds the agentic layer on top. If the proposal quietly assumes a greenfield rebuild, that's a cost and a risk nobody priced.

3. Will your team own it when they leave?#

This is the question that separates a transfer from a disguised "buy." A vendor wants you dependent, because dependency is recurring revenue. A transfer partner wants you independent, and designs the engagement to get you there. The test is simple: can your developers run the agents without them in six months? Ask it directly, and make capability transfer a written deliverable rather than a hope. We wrote about why this is the real build-vs-buy decision in build vs buy your AI capability.

4. Do they augment your developers, or replace them?#

Watch how a partner talks about your team. If the pitch is "replace expensive engineers with agents," two things are true: your best people will resist it (quietly killing the rollout), and you're buying a fragile system with no one who understands it. The approach that holds up turns your developers into the operators of the agents, so the gains are theirs and the knowledge stays in the building. We make the case for that in augment, never replace.

5. Can they show production, not demos?#

A demo proves a model can do something once, under ideal conditions. Production is the hard part, and it's where most AI initiatives quietly die (we covered why in why AI demos die before production). Ask a partner what they run in their own production, today. A studio that ships its own software with agent fleets is showing you the capability directly. A partner who only has demos and case decks is asking you to fund their first real attempt.

6. Are they honest about when you don't need them?#

A good partner will tell you when the work isn't worth doing, or when your team could handle it in-house. That honesty is a signal, not a weakness, it means they're optimizing for your outcome rather than the contract. A partner who finds a reason you need every service they sell is selling, not advising.

How we measure up#

We built this checklist around how we work, so we'll be direct about it. We're a dev studio that runs agent fleets in production to ship our own software, and we install that operating model on client teams, on their stack and standards, then train their developers to own it. Augment, never replace. Independence, not dependency. And when a team doesn't need us, we say so.

If you're vetting partners and want one who'd pass their own checklist, that's the conversation to have. Talk to us about your team.

Common questions

How do I evaluate an AI consulting partner?

Score them on capability transfer, not the pitch. Ask whether they build or only advise, whether they work on your existing stack or impose a rebuild, whether your team owns the result when they leave, whether they augment your developers or replace them, and whether they can show production work rather than demos. A good partner makes you independent; a weak one makes you dependent.

What questions should I ask an AI agency before signing?

Ask to see agents they run in their own production, not a demo. Ask who owns the capability in six months, you or them. Ask whether they work inside your codebase and standards. Ask how your developers come out of the engagement more capable. If the answers are vague or all about their platform, that's the answer.

What's the difference between an AI strategy firm and a build partner?

A strategy firm defines what to do and hands you specs; the production gap is your problem. A build partner gets agents running on your code and reviews real failure modes with you. For getting AI into production, the second is what moves the needle. Strategy without a build rarely survives contact with your repo.

How do I avoid getting locked into an AI vendor?

Make capability transfer an explicit deliverable, not a hope. Ask whether your team can run the agents without the vendor after the engagement, and write that into scope. A vendor that wants you dependent will resist; a transfer partner will design for it.

Want this running on your team?

Get your assessment