How to Choose an AI Automation Agency: 12 Questions and 7 Warning Signs

Ask one question first: name a system you built that is still running in production today, and the number it moved. An agency that has shipped answers in a sentence, with a client, a process and a figure. An agency that has not will pivot to model comparisons, a strategy framework, or a demo video. That single question filters out most of the market before you get to the details below.

The 12 questions, and what a good answer sounds like

1. Which single process will you automate first, and what saving do you expect? Good: "Quote follow-up. You send roughly 60 quotes a month and follow up manually; we expect to recover 6-8 hours a week and lift follow-up rate to 100%." Bad: a list of six workstreams starting in parallel.

2. Who owns the code, the prompts, and the API accounts afterwards? The only acceptable answer is: you do, and the accounts are in your name from day one. Anything else means you cannot leave without rebuilding.

3. Which model and vendor, and what if they raise prices or deprecate it? You want a named model, a reason for the choice, and an abstraction layer so a swap is a config change rather than a rebuild. Models get deprecated on a 6-18 month cycle. Ask who pays for that migration.

4. Where does our data go, and is it used for training? Enterprise API tiers and Azure OpenAI do not train on your data; consumer tiers can. For sensitive data you want the data flow documented in the contract, plus a named region.

5. Will it work inside the tools we already run, or do we move to your platform? If the automation only lives in their account or their product, your process is now hostage to your vendor relationship.

6. What is the monthly running cost after launch, separate from your fee? A real answer is a range with volume assumptions: "about EUR 60-120 a month at 1,000 conversations". No answer means they have not modelled your volume.

7. Is there a pilot before a large commitment? One narrow use case, 2-4 weeks, a fixed price, and a measured result. Any agency confident in its work will accept being judged on a small piece first.

8. What happens when the AI is wrong? You want three things named: a human review step on consequential decisions, monitoring with alerts, and a documented fallback. "Our accuracy is 95%" without a review queue is an unmanaged 5%.

9. Who exactly does the work? Ask for the names and the split between the person in the meeting and the person writing the code. Subcontracting is fine; undisclosed subcontracting is not.

10. What does handover look like? Repository, prompt library, workflow exports, an architecture note, and 4-10 hours of training. Get it listed in the quote, not promised verbally.

11. How will we measure success in 90 days? One or two numbers agreed before the build starts: hours saved per week, response time, qualification rate, cost per processed document. If the metric is invented after launch, it will be a flattering one.

12. What have you failed at? Every agency with real production experience has a project that underdelivered and can say why. A spotless record after two years of AI work means very few projects.

The 7 warning signs

1. Demos only, no production references. A polished demo takes a day to build and proves nothing about maintenance, edge cases, or integration.
2. They lead with the model, not your process. Which LLM is a ten-minute decision. Where your hours are going is the project.
3. A licence dressed as a build. Monthly fee, no handover, no ownership. That is a product subscription. It can be the right choice, but price it as one.
4. No pilot offered, only a 6-month contract.
5. Hourly billing on a first build with no cap. You are funding their learning curve with unlimited downside.
6. No mention of data handling until you raise it. Anyone who has done AI work for a mid-sized company brings up GDPR unprompted.
7. Guaranteed percentages. "We will cut your costs by 40%" before seeing your processes is a sales figure, not an estimate.

A scoring sheet you can take into the meeting

Score each 0-2 and total it. Below 14 out of 24, keep looking.

Score each criterion 0 for the weak answer, 1 for the partial answer, 2 for the strong answer.

1. Production references. 0: none. 1: demos only. 2: named, running, with a number.
2. Ownership of code and accounts. 0: theirs. 1: mixed. 2: yours.
3. Integrates with your existing stack. 0: no. 1: partly. 2: yes.
4. Pilot offered. 0: no. 1: vague. 2: fixed scope and price.
5. Running cost transparency. 0: no answer. 1: rough. 2: modelled on your volume.
6. Error handling and human review. 0: not mentioned. 1: mentioned. 2: designed in.
7. Named success metric. 0: none. 1: soft. 2: agreed and measurable.
8. Handover and training in the quote. 0: no. 1: verbal. 2: written.
9. Data handling and GDPR. 0: avoided. 1: on request. 2: documented in the contract.
10. Model-change plan. 0: none. 1: "we'll see". 2: an abstraction layer plus who pays.
11. Named delivery team. 0: no. 1: partial. 2: yes.
12. Honest about a past failure. 0: no. 1: deflects. 2: specific.

Agency, consultant, or an in-house hire?

A consultant delivers a recommendation; you still have to build it. An in-house AI engineer costs EUR 55,000 or more in first-year salary and needs 3-5 months to become productive, which pays off only with a continuous pipeline of AI work. An agency ships the first working system in 2-6 weeks at project cost. The honest rule: use an agency until AI becomes a full-time function in your company, then hire, and keep the agency for the parts your hire does not cover.

What to do this week

Shortlist three agencies. Send all three the same one-page brief describing a single process and ask for a fixed-price pilot quote against it. The differences in what comes back, in what they ask you before quoting and in how specific their numbers are, will tell you more than any capability deck.

Related reading

• AI Agency Pricing in 2026: What You Actually Pay, Line by Line
• What Is an AI Agency? A Guide to AI Automation for Business
• AI Automation Costs and ROI for Small Businesses

Want to compare these criteria with a real delivery process? Review our AI automation agency services and implementation approach.