How to Build an AI Voice Agent for After-Hours Customer Support

Calls don’t stop at 6 p.m. In Zendesk’s CX Trends 2026 research, 74% of consumers said they now expect service around the clock. Staffing a night shift for a handful of calls rarely pays, and voicemail loses the customer. A voice agent fills that gap. Our Agent Engineering team builds them, and this guide walks through the design that holds up in production.

Five words explained

• Speech-to-text (STT): turns the caller’s voice into text, live, as they speak.
• Text-to-speech (TTS): turns the agent’s reply into a natural voice.
• Telephony: the phone side: the number, call routing and transfers, usually through a cloud phone provider.
• Barge-in: the caller interrupting the agent mid-sentence, which the agent must notice and stop.
• Latency: the gap between the caller finishing a sentence and hearing the reply.

What do after-hours callers need from an AI voice agent?

Before choosing any technology, sort your after-hours calls into three groups. Pull a month of call logs or voicemails and tag each one.

1. Urgent: a real emergency that can’t wait until morning, such as a system outage, a safety issue or a blocked payment. The agent’s job is to confirm it fast and reach the on-call person.
2. Routine: questions or tasks it can finish alone, such as order status, opening hours, booking or moving an appointment.
3. Capture: everything else. The agent takes a clear message, creates a ticket and books a call-back slot.

The three groups become the voice agent’s job description. Anything not on it goes to group 3.

How does an AI voice agent work?

An AI voice agent passes every call through four layers: telephony, a speech pipeline (speech-to-text, the AI agent, text-to-speech), tools into your systems, and a hand-off to a person. Keep the layers separate, so you can swap a provider without rebuilding the rest.

The call travels left to right through the speech pipeline and the reply comes back over the top. The agent only touches your systems through its tools, and urgent calls leave through the phone layer to a real person.

Should a voice agent be cascaded or speech-to-speech?

There are two ways to build a voice agent’s speech layer.

• Cascaded: three separate steps, speech-to-text, then the language model, then text-to-speech. You get a written transcript of every turn, you can test each step on its own, and you can swap any part.
• Speech-to-speech: one model listens and speaks directly. It responds faster and handles interruptions more naturally, but you see less of what happened inside, and you are tied to that one model.

For support calls, start cascaded. Transcripts make testing, auditing and fixing far easier, and voice-AI testing firm Coval reports cascaded is still the default for enterprise voice agents in 2026 for exactly that reason. Revisit speech-to-speech if speed becomes your main complaint.

How fast must an AI voice agent respond?

An AI voice agent should start replying about one second after the caller stops speaking. Speed decides whether a voice agent feels helpful or broken. Twilio’s latency guide sets a starting target of about 1.1 seconds from the caller going quiet to hearing a reply, with 1.4 seconds as the upper limit. Most of that budget goes to speech-to-text and the model’s first words. How to stay inside it:

• Keep every service in one region, close to your phone provider. Each extra network hop adds delay.
• Stream everything. Start speaking the first sentence while the rest is still being written.
• Use a fast model for the conversation. Save a slower, stronger model for rare, hard steps.
• Say something while tools run. “Let me check that booking for you” covers a two-second system lookup.
• Keep answers short. One or two sentences, then a question. Long answers belong in an SMS.

What happens on an after-hours call?

On an after-hours call, the voice agent greets and identifies the caller, checks for urgency first, then either wakes the on-call person, resolves the request, or takes a message and books a call-back.

The urgency check comes first, so an emergency never waits behind a booking question. A call the agent can’t finish never ends in a dead end: it becomes a ticket with a call-back time the caller has agreed to.

What details make or break an AI voice agent?

• Confirm what you heard. Read back names, dates, amounts and reference numbers. Speech-to-text gets these wrong more often than ordinary words. For long account numbers, let callers type them on the keypad.
• Check identity before sharing anything personal. Use the same checks your human team uses, such as a date of birth and a one-time code sent by SMS.
• Say it is an AI, and say if you record. Recording and consent rules vary by country, so agree the wording with your legal team.
• Write the escalation rule down. “Wake the on-call engineer if a customer reports a full outage” is a rule. “Escalate if it seems serious” is not.
• Hand over with context. When the agent transfers or creates a ticket, it passes the transcript and a two-line summary, so nobody asks the customer to repeat themselves.
• Plan for languages and accents your customers actually use, and test each one.

How do you test an AI voice agent?

Test an AI voice agent the way you’d test a new hire on the phones: with realistic calls. Build a set of 100 or more call scripts from real after-hours calls, then play them through the agent with simulated callers, background noise and different accents. Run each script several times, because answers vary. Track five numbers: share of calls resolved without a person, correct escalations, missed escalations (which should be zero), errors on names and numbers, and slow replies above your latency limit. How to Test AI Agents covers the full method.

How do you run a voice agent after launch?

Every voice agent call pays for phone minutes, speech-to-text, text-to-speech and model tokens, so set a cost-per-call target before go-live; the cost of running AI agents shows how to model it. Treat prompts, the voice, the speech providers and the model version as one release bundle, roll out changes to a small share of calls first, and score a sample of real calls every day, as described in DevOps for AI agents.

Common mistakes

• Rebuilding the old phone menu with a nicer voice.
• No warm transfer to a person, so urgent callers hit a dead end.
• Reading long answers aloud instead of sending them by SMS.
• Testing only with clean audio in a quiet office.
• Spreading speech, model and phone services across different regions.

How SHVX can help

If your after-hours calls go to voicemail, you are losing customers while you sleep. Our Agent Engineering team designs and builds voice agents with the escalation rules, identity checks and testing built in, and our Agent Operations team runs them after launch.

Book an assessmentinfo@shvx.in

Learn more about Agent Engineering.

FAQ

How fast should an AI voice agent respond?

Aim for replies that start in about one second after the caller stops speaking. Much slower and callers start talking over the agent or hang up.

Can a voice agent transfer to a human?

Yes. A warm transfer passes the call and a summary to a person; if nobody is available, the agent books a call-back.

Do I need to tell callers they are talking to AI?

Yes. It builds trust, and rules in many countries require it. Check local recording and consent rules too.

Which is better for support, cascaded or speech-to-speech?

Start with cascaded for its transcripts and testability. Consider speech-to-speech when speed matters more than visibility.

Latency targets checked against Twilio’s published guide in November 2026.