A voice agent that answers the phone, holds a natural conversation and books the appointment sounds like science fiction that has finally arrived. In the narrow cases where it works, it genuinely has. In the others, it is a very expensive way to annoy people.
Here is the picture I give clients before they spend anything.
How a voice agent works
It is a chain of four things, and each one adds delay and cost:
- Telephony. A phone number and the connection, from a provider such as Twilio.
- Speech to text. What the caller says is transcribed as they speak.
- The language model. It reads the transcript and decides what to say or do, including calling tools such as a calendar or CRM.
- Text to speech. The reply is turned back into a voice and played to the caller.
Platforms such as Vapi bundle these steps so you are not stitching them together yourself, and an orchestrator like n8n connects the agent to your real systems: checking availability, writing to the CRM, sending a confirmation. The thinking behind those tools is the same as for a text assistant, covered in AI agents vs AI automation.
Where they genuinely help
- After-hours calls. Capturing the enquiry at 9pm that would otherwise go to voicemail and never be returned.
- Appointment booking and reminders. A bounded task with a clear goal and a checkable result.
- First-line lead qualification. Asking the same four questions every time and routing the promising callers to a person. See AI lead qualification for how to score them.
- Overflow. Picking up when every human is busy, instead of ringing out.
- Simple information. Hours, location, pricing ranges, what to bring.
The pattern: narrow, repeatable, low-stakes, easy to hand over.
The limits nobody mentions
Accents and background noise
Speech recognition is much better than it was, but it is not equally good for every accent, line quality or noisy environment. Test with the voices your real callers actually have, not the clean audio in a demo.
Interruptions and tangents
People talk over each other, change their minds mid-sentence and wander off topic. An agent that handles a polite scripted call perfectly can fall apart on a real one.
Complex or emotional calls
A complaint, a bereavement-related query or anything that needs judgement and empathy is the wrong job for an agent. Detect it early and transfer.
Cost adds up per minute
Every stage of the chain is billed, usually by usage. A ten-minute call costs far more than a ten-message text exchange. Model your cost at real call volumes before you commit.
Why speed decides everything
In text, a two-second delay is invisible. On the phone, a pause of that length feels broken. Callers start repeating themselves or hang up. Latency is the main thing that separates a voice agent people tolerate from one they like.
Practical consequences: use faster, smaller models for the conversation itself, keep tool calls to what is necessary, and keep the agent's instructions short. A clever answer that arrives late is worse than a plain one that arrives on time.
Consent, disclosure and the law
Treat this as a real legal question, not a footnote. Rules on recorded calls, automated calling, marketing calls and disclosing AI differ widely between countries and sometimes between regions. I am not a lawyer and this is not legal advice. Check the rules where you operate and where your callers are.
- Say it is an AI, early and plainly.
- Tell people if the call is recorded, and why.
- Get consent before outbound calls, and honour do-not-call lists.
- Let anyone reach a human, every time, on request.
- Protect the data. Call transcripts are personal data. Know where they are stored and who can read them.
How to start safely
- Begin with inbound, after-hours calls only. No outbound until you have the consent question settled.
- Pick one job, such as taking a message and booking a slot.
- Test with at least thirty real recorded calls from your own customers.
- Review every transcript for the first fortnight.
- Make the human handover work flawlessly before anything else.
The reason I am cautious is the same reason I recommend a staged rollout for every agent: voice makes the mistakes more visible and harder to undo. Build the narrow version, watch it fail in small ways, fix those, and only then widen it.