How to Vet an AI Automation Agency in 2026 (12,000 Agencies, Most Have Never Shipped)

There are over 12,000 self-described AI automation agencies as of mid-2026. The majority launched in 2024 or 2025 after watching YouTube tutorials about ChatGPT and Zapier. Many have never shipped a system to a real client. Some have shipped one or two small projects. A small minority have the depth to handle production systems for growing service businesses.

The 10-point checklist below will identify which category any agency falls into within one discovery call. Ask these questions and listen carefully to how they answer. Evasion, jargon, and credential-dropping without specifics are disqualifiers.

We apply this same checklist to ourselves. If you asked us these questions, we could answer every one with specifics. That is the bar.

Red Flags That Disqualify an Agency Immediately

Before the 10-point checklist, these are automatic disqualifiers. Any one of these should end the conversation.

They cannot explain what they built without jargon: Ask "Tell me about the last system you shipped. What did it do? What was the result?" If the answer is full of terms like "synergistic AI ecosystem" without concrete specifics, they have not shipped anything real.

They charge for a "discovery call": A sales call is free. An actual analysis of your business is worth charging for. If they call it discovery but it is really a 30-minute pitch, that is a misrepresentation.

They promise results in weeks for complex systems: Production-grade AI systems with real integrations take 2 to 8 weeks to build properly. Anyone promising a complete lead machine in 72 hours is delivering a template, not a custom system.

They cannot name the actual tools they will use: Ask "What automation platform do you build on? What AI model? What telephony provider?" Legitimate agencies have specific answers. Non-answers mean they will figure it out after you pay them.

They have no client references: Not testimonials on their website -- actual references you can call. A one-year-old agency with no referenceable clients has not delivered anything worth referencing.

Their pricing is "custom" for everything: Agencies with real productized services know what things cost because they have built them before. Agencies that quote everything as "custom" are figuring out pricing as they go.

The 10-Point Vetting Checklist

Work through these questions in your first call. Take notes on the answers.

1. Show me a live system you built for a client in my industry. Acceptable: A live link, a screen share, a recorded demo. Not acceptable: "We have done similar work" without specifics.

2. What was the measurable business outcome for that client? Acceptable: "They recovered 12 leads per month, which at their close rate and average job value equals $X in additional revenue." Not acceptable: "The client was happy with it."

3. What tech stack do you build on? Acceptable: Specific tools -- n8n, Vapi, Twilio, Retell AI, Claude API, Resend, Supabase. Not acceptable: "Whatever is best for your situation."

4. Who will actually build my project? Acceptable: Named people with verifiable experience. Not acceptable: "Our team" without being able to identify who that team is.

5. What does post-launch support look like? Acceptable: Defined window (30-day included, then retainer options). Not acceptable: Vague reassurances about "being there for you."

6. What is your process when something breaks after launch? Acceptable: SLA commitments, escalation paths, monitoring tools they use. Not acceptable: "We will fix it" without specifics.

7. What does your discovery process look like? Acceptable: A defined output (roadmap, ROI projections, prioritized recommendations). Not acceptable: "We will get to know your business."

8. Can you provide two or three client references in my industry? Acceptable: Names and contact information for clients who will take your call. Not acceptable: Names with no contact information.

9. Have you personally run a business in my industry or a similar one? Acceptable: Direct operator experience. Not acceptable: "I consulted for clients in that industry."

10. What will you not do for me? Acceptable: Honest scope limitations. Not acceptable: "We can do anything."

What Should Good Answers Look Like?

For each of the 10 questions, here is what a credible answer looks like versus a non-answer.

Question 1 (Show me a live system): Good answer: "Here is a screen recording of the call handling system we built for a plumbing company. It answers after-hours calls, qualifies leads, and books appointments directly into ServiceTitan." Non-answer: "We have done work in the trades space and would be happy to build something similar for you."

Question 4 (Who will build it): Good answer: "Barrett will handle the strategy and system design. Here is the LinkedIn profile." Non-answer: "We have a team of experienced developers."

Question 8 (References): Good answer: "You can call Maria at Sunshine HVAC -- 813-555-0100 -- she was our first client and has been on managed ops for 18 months. She will tell you what she actually thinks." Non-answer: "We have a lot of happy clients. Here are some reviews from our Google listing."

Question 9 (Operator experience): Good answer: "I ran a home services company for 6 years before starting this agency. I know exactly what a dispatcher's day looks like and where the calls fall through." Non-answer: "I have worked with a lot of home service clients."

The pattern: good answers are specific, verifiable, and honest about limitations. Non-answers are vague, jargon-heavy, and designed to avoid commitment.

How Do You Evaluate Pricing Proposals?

Once you have narrowed to two or three agencies using the checklist, you will receive proposals. Here is how to evaluate them.

Scope clarity: Does the proposal define exactly what is being built, what integrations are included, and what is explicitly not included? Vague scope is the source of most scope creep and cost overruns.

Deliverable specificity: "AI automation system" is not a deliverable. "n8n workflow that intercepts missed calls, runs Vapi AI conversation, pushes lead data to ServiceTitan, and triggers Resend follow-up email within 5 minutes of call end" is a deliverable.

Timeline realism: Two-week timelines for complex systems are unrealistic. Three to six weeks for a properly built, tested, production-grade system is normal.

Payment structure: 50% at kickoff, 50% at delivery is standard and fair. 100% upfront is a red flag.

Post-launch definition: What specifically is included in post-launch support? How long? What is the escalation path for issues?

References and guarantees: Will they let you talk to references before you commit? Compare proposals on scope specificity, not total price. The cheapest proposal often wins on paper and loses in delivery.

How to Think About Agency vs. Freelancer vs. DIY

Your three options for AI automation are a full-service agency, a freelancer from a platform like Upwork or Fiverr, or building it yourself with tools like Zapier or n8n. Each has a legitimate use case.

Freelancers work for simple, well-defined projects with low stakes. Need a Zapier workflow that moves data from one place to another? A $200 Upwork hire might be perfect. Need a production AI voice system handling your after-hours emergency calls? A freelancer is the wrong call. No monitoring, no support, no one to call at 9 PM when the system starts failing calls.

DIY tools (Zapier, Make, n8n) work if you have the technical patience to learn them and the time to maintain them. The true cost calculation needs to include your time. If you spend 15 hours per month on automation maintenance at a $200 effective hourly rate, that is $3,000 per month in opportunity cost.

Full-service agencies make sense when the automation is revenue-critical, complex, or when your time is better spent running your business than maintaining software. The right agency delivers a system that works, documents it, and maintains it so you do not have to think about it.

The red flag scenario: business owner hires cheap freelancer, gets a barely-working system, spends months troubleshooting it, finally gives up, then pays an agency to build the right system from scratch. Total cost: 2x. This is extremely common.