Answer
What should I test in an AI receptionist demo before I buy?
Last updated
Why does an AI voice demo sound good and then fall short?
Our first client ever cancelled on us, and it had nothing to do with price. The AI was calling their existing customers, with their consent, to upsell them, and some of those people picked up the phone in the wrong place: somewhere with someone talking in the back, or a humming sound, or just out on the street. The AI could not tell those sounds apart. It got lost, or it stayed silent, because it is built to wait politely for the caller to finish a sentence, and a street does not let you finish one.
The other half of the story was our native Lithuanian, which the AI could not deliver reliably either as well as the noise filtering, and that is why they cancelled. It was a hard lesson, and I would rather you got it from me than from your own customers: an AI can sound great in a demo for a few sentences, and then in production a lot of things break.
None of that should be a surprise, because building the demo is the easy part. Anyone can have something answering the phone and saying a couple of sentences nowadays. You can ask a chat tool how to do it, and you may well be approached by companies, or by your own IT team, who offer to take care of it for you. What none of them can show in a demo is the street.
Why do cheaper vendors leave buyers stuck?
We have a lot of competitors that are cheaper than we are, and when clients buy from them, this is what I see. The project stalls for a couple of months, the clients get burned, and they often cannot simply go and buy from somebody else, because they signed an agreement with that first vendor.
If you ask how it is going, the answer sounds like this: yes, it is taking half a year to deliver, yes, it is still in the demo phase, yes, nothing is rolled out. So before you choose on price, think about the opportunity cost, and what picking the wrong vendor really costs you. It is not about the price.
What should I ask a voice AI vendor first?
When a voice system scales up to production, a few variables decide whether it works. Noise comes first, and it is the first bottleneck you will meet. The second is context. If the AI is prompted wrong it gets confused, does things nobody intended, or misunderstands what the caller meant.
Here is the example I like best, because every sales team has lived it. The AI calls a person and the person says they are busy.
An AI prompted wrong
Sorry, we are busy right now.
Flags the lead as a rejection: not interested in the service.
An AI prompted right
Sorry, we are busy right now.
Schedules a callback in a couple of hours or a couple of days. Yes, that is possible, in case you were wondering.
Both heard the same four words. One lost you a lead and the other one kept it. That is why there are two parts to getting this right. Programming is the prompt. Reliability is the functions, which are the tools the AI is given: booking a consultation, adding people to a do-not-call list, transferring a call to a human.
So the first thing to do with any partner is ask about the technical side, and these four are the ones I would not let them skip.
- 01
Noise filtration
What happens when the caller is in the street, or someone is talking behind them, or there is a hum?
- 02
Concurrent calls
How does it behave when several calls are live at the same time?
- 03
Callbacks
Does a busy caller get a scheduled callback instead of a rejection?
- 04
Transfers to a human
Can it hand a call over to a real person?
What is a paid pilot, and how do I run one?
What we propose to most of our clients is a first paid pilot. You see the system working on one mapped workflow, and then your job is to test it hard: try to break it, play with it, try to get it off topic, and see whether it stays consistent and reliable.
It costs time as well as money, and people tend to forget the first part. A big chunk of your time, or your team's, goes into mapping everything out and working out what you actually want. You might start with one use case and find after testing that it needs to do something different or better. There is a lot of going back and forth, and that is normal, so do not read it as a sign that something is wrong.
How do I map my calls before I talk to vendors?
Download the free prompt linked in the video's description and drop it into the AI chat you already use. It asks about the bottlenecks in your business and your repetitive scenarios, so you can come up with what you need.
The real goal is a process tree: what happens to a lead when the phone rings. It is a decision tree, so if a person says this, then that should happen, and if they say something else, then something else happens, and all of it goes into your agent.
Reliability also needs a review system, and it is fair to ask your supplier whether they have one. One option is an agent that reviews each call. Another is a call transferred to a person, who gets an email after the call and reviews it.
What does the wrong supplier cost, and what do I get when it works?
If you pick the wrong supplier the opportunity cost is huge, especially if you prepay, which is exactly why we always suggest a paid pilot first and a full rollout only when everything is okay. Armed with the questions above, you can walk into that sales call with a little healthy skepticism, or just send them the questions in an email.
On the other side of that tunnel there is something good. You get a system trained on your best practices, one that is always consistent and always reliable, and that fits like a glove. You do not have to learn a new CRM or any other new tool, because it integrates into the ones you already use. A simple dashboard lets you review the calls if something goes wrong, although your AI partner should probably do that for you.
From there, voice AI can call your leads, handle incoming traffic, take several calls at once and do consistent follow-ups and reminder calls. There is a lot of opportunity in it, and your fantasy is the bottleneck.
So the next time someone plays you a smooth demo, ask them what happens when the person on the line is standing in the street. If the answer is a good one, you may have found a real partner. If it is silence, you have just heard what our first client heard.
Short answer
Test what a demo cannot show: how the system copes with noise, concurrent calls, callbacks and transfers to a person. A demo only has to sound good for a few sentences, and in production a lot of things break. Ask the vendor about those four things first, then run a small paid pilot on one mapped workflow, try to break it, and roll out fully only if it stays consistent.
Justas Butkus is a fractional AI officer based in Vilnius, Lithuania, working with mid-market companies and scale-ups across the UK, EU and US at two to four days a month. He builds and operates the production AI systems he advises on, and takes no commission, referral fee or revenue share from any vendor.
Want to test a voice AI setup on your own calls?
Book a call and we will map your calls and look at a first paid pilot together. The phone agents run on Ainora, the voice platform of my company AINORA, MB.
Frequently asked questions
Is it hard to build an AI voice demo?
No. A demo that answers the phone and says a couple of sentences is extremely easy to build. The difficulty starts when it meets real callers in production.
What is the first thing that breaks in production?
Noise. It is the first bottleneck you run into: other voices, humming, the street. The AI gets lost or stays silent.
What do the functions of a phone AI do?
They are the tools the AI is given, such as booking a consultation, adding people to a do-not-call list and transferring a caller to a human.
Should I decide on price?
No. Compare the technology. A wrong supplier costs far more in opportunity cost, especially if you prepay.
Do I need a new system to review the calls?
No. A simple dashboard, an agent that reviews each call, or an email to the person responsible after a transferred call is enough.