Module 13 · Lesson 4 of 7Free

Voice agents: real-time spoken conversation

Level

Advanced

Topic

Multimodal and voice applications

Reading time

17 min

Eight seconds of silence on a phone line

Deskora is a help desk app. A shop’s support team uses it to answer customer emails and chats from one shared inbox, and each customer problem becomes a ticket. Priya is a support agent at a kitchenware shop on Deskora. Tomás is the Deskora engineer who builds its AI features.

Deskora Chat already answers shoppers on a shop’s website with no support agent in between: help-center answers and verified order status, everything else to Priya. Shops now want the same on their phone line, Deskora Voice.

Tomás’s first test reuses Chat’s safe path. A caller named Amara asks, “Has my order shipped?” Chat checks the question, drafts a reply and checks the whole reply before letting it out. On the phone that is about 8 seconds of silence. After two seconds Amara says “Hello?”, and the system treats that as a new question.

So the question for this lesson is: how do you make a model answer a phone call fast enough to feel like a conversation, without loosening any of Deskora Chat’s limits?

Amara, the call volumes and every stage time in this lesson are this lesson’s scenario and estimates. Chat’s stage times come from the Deskora Chat design.

One spoken turn passes through six stages

A spoken assistant like this is often called a voice agent, even when it takes no actions. Here is one turn.

Voice activity detection (VAD) is a small model that answers one question about each slice of audio: is someone speaking? Turn detection builds on it to decide the caller has finished. In LiveKit Agents, a turn detector model predicts the end of a turn from both the meaning and the sound of the speech, on top of VAD ^{3Turns overviewLiveKit, LiveKit Agents documentation}. Speech to text and text to speech covers the two speech services.

The point is overlap: earlier parts of a reply are spoken while the model writes later parts ^{2Overview of PipecatPipecat, Pipecat documentation}, and speech to text works while Amara talks. How much time does one turn have?

The budget: about 1.5 seconds from silence to voice

In a study of ten languages, people typically started replying within about 200 ms of a question ending ^{1Universals and cultural variation in turn-taking in conversationTanya Stivers, N. J. Enfield, Penelope Brown and others, Proceedings of the National Academy of Sciences, 2009}. A machine must wait for silence first, so it can’t match that. Pipecat’s docs say typical voice interactions complete in 500 to 800 ms ^{2Overview of PipecatPipecat, Pipecat documentation}.

Deskora’s goal (an estimate for this scenario): first sound within 1.5 s of the caller’s last word on a typical turn, and never 2 s of silence. If a reply is late, code plays a short fixed phrase.

One help-center turn, timed from Amara’s last word, at the course’s assumed 100 output tokens a second:

The 1.3 s, 3.7 s and 2.1 s come from the Deskora Chat design; the other times are this lesson’s assumptions.

Each cut moves one row:

• Stream everything, and speak on the first sentence. Text to speech starts after about 15 tokens, not after a 3.7-second reply.
• Lower effort. At low effort, adaptive thinking may skip thinking on easy inputs ^{8Extended thinkingAnthropic, Claude API documentation}; usage.output_tokens_details.thinking_tokens reports what there was. 200 thinking tokens cost about $0.0001, but 2 seconds of silence.
• A shorter turn wait. With a turn detector model, the silence wait can drop from 0.5 s to about 0.3 s.
• The classifier beside the draft. Code holds the first sentence until the input classifier passes.
• A short, cached prompt shortens prefill; see Caching for LLM apps. Claude Haiku 5.5 is already the small model.

One more cut reaches about 1.3 s. LiveKit calls it preemptive generation: the model starts on the interim transcript during the turn wait (on by default in livekit-agents 1.8.10). If the final words differ, the early call is redone. Order-status turns skip retrieval and reach about 1.0 s.

Where the time goes in an LLM request teaches this stage model. But streaming breaks Chat’s rule that every check finishes first.

Decision: keep Chat’s checks without the 8-second hold

Design: a safe customer-facing assistant held each reply and checked it whole, since a few seconds behind a typing indicator was acceptable. A phone has no typing indicator.

Speak each sentence as soon as it passes

The say function hands one sentence to a streaming text-to-speech service and returns once it is queued.

import re
from collections.abc import Awaitable, Callable

import anthropic
from anthropic.types import MessageParam

client = anthropic.AsyncAnthropic()  # reads ANTHROPIC_API_KEY from the environment

SENTENCE_END = re.compile(r"(?<=[.?!])\s+")
PROMISE = re.compile(r"\b(I|we)( will|['’]ll| can) (refund|replace|send|cancel|give)", re.I)
LONG_NUMBER = re.compile(r"\d{3,}")
HANDOFF = "I can't sort that out on this call, so I'm passing you to the team."


def is_safe(sentence: str, allowed_numbers: set[str]) -> bool:
    if PROMISE.search(sentence):
        return False
    return all(n in allowed_numbers for n in LONG_NUMBER.findall(sentence))


async def speak_reply(system: str, messages: list[MessageParam], allowed_numbers: set[str],
                      say: Callable[[str], Awaitable[None]]) -> list[str]:
    """Stream one reply and speak each sentence that passes. Returns what was sent to speech."""
    sent: list[str] = []
    buffer = ""
    async with client.messages.stream(
        model="claude-haiku-5-5",
        max_tokens=400,  # thinking counts toward this too
        system=system,
        messages=messages,
        output_config={"effort": "low"},
    ) as stream:
        async for text in stream.text_stream:
            buffer += text
            *sentences, buffer = SENTENCE_END.split(buffer)
            for sentence in sentences:
                if not is_safe(sentence, allowed_numbers):
                    await say(HANDOFF)
                    return sent + [HANDOFF]
                await say(sentence)
                sent.append(sentence)
    if buffer.strip():
        last = buffer.strip() if is_safe(buffer, allowed_numbers) else HANDOFF
        await say(last)
        sent.append(last)
    return sent

allowed_numbers holds the verified order number and the numbers in the retrieved articles. The call loop runs speak_reply as an asyncio task and transfers the call if the last item is HANDOFF. When Amara talks over Deskora, the loop cancels the task and stops text to speech. Cancelling leaves the async with block, which closes the stream.

One run might send “Your order shipped on Monday.” and “It should arrive by Thursday.” as two sentences.

That reply is an example; model output differs between runs. On Claude Haiku 5.5, as of October 2026, one voice turn of about 3,600 input and 40 output tokens costs about $0.0004 ^{7PricingAnthropic, Claude API documentation}.

A phone call has rules a chat doesn’t

A call never pauses, so the voice server needs a rule for each of these moments.

After a barge-in, the history must hold only what Amara heard. Text to speech reports how much audio it played, and the frameworks cut the history to match.

Keypad digits arrive as DTMF tones, the beeps a phone sends for each key, and they are exact. A transfer comes with a ticket holding the transcript, so Amara doesn’t repeat herself; Human in the loop covers what a good hand-back contains.

Same limits, new channel: safety on the phone

• The AI notice comes first, as recorded audio. Fixed words, approved by Hana, the shop’s manager: this is the shop’s AI assistant, what it can’t do, and that pressing 0 reaches a person. The model never writes it. AI regulation covers the law.
• Recording needs consent. Rules differ by country, and some places need everyone’s agreement. The notice says the call is recorded and why, and each shop can turn recording off. This is not legal advice.
• Verify before any order status. Chat’s rule: order number and the order’s email, compared in code, one reply for every failure, three tries, then a hand-off. The number comes by keypad; the spoken email is read back first. Caller ID is never proof, because it can be faked.
• No actions. The model has no tools. Code gives it the verified order as a document. Refunds, replacements and complaints go to Priya.
• Mask what callers say aloud. Callers read card numbers out, so the transcript is masked before the model, the logs or the ticket.

Every one of these rules works on text, which is the main argument against the alternative.

Speech-to-speech models skip the text

A speech-to-speech model takes audio in and gives audio out, with no separate speech steps. Google’s Gemini Live API is one: it streams audio both ways over a stateful WebSocket, supports barge-in and function calling, and returns transcripts of both sides ^{4Gemini Live API overviewGoogle, Gemini API documentation}. The Claude API documents no audio input as of October 2026.

Frameworks and the phone network

Turn detection, barge-in and history trimming are hard to build by hand. Two open-source Python frameworks do them, described here from their docs and source:

Either takes the same Claude call and sentence gate; choose on the phone connection and turn controls you need.

Telephony in one paragraph. A shop’s number lives with a telephony provider. When Amara calls, the provider streams the audio to Deskora over a WebSocket, or connects the call through a SIP trunk, a standard link between phone systems and software. Phone audio is narrowband: Pipecat’s Twilio serializer uses 8,000 samples a second, so pick a speech-to-text model that handles phone audio. The provider bills by the minute.

What one call minute costs

The unit is one completed call, priced by the minute. Per caller turn it counts the input classifier, retrieval on half the turns, the drafting call and the output classifier. Not counted: hosting the voice server and Priya’s time after a hand-off.

Prices as of October 2026: Twilio US, Deepgram pay-as-you-go, Claude Haiku 5.5 at $0.10 per million input tokens and $0.50 per million output tokens.

Call length, turns per minute, speaking time and 1,000 calls a day are this lesson’s estimates.

The model is under a tenth of the minute; text to speech and telephony are over half. So the levers are shorter spoken replies and the number type, not a cheaper model. A call costs about ten times a Deskora Chat conversation ($0.011). Deepgram also sells a bundled voice agent at $0.075 a minute, billed on connection time ^{5PricingDeepgram}.

Where this shows up

• Lab: a voice agent with a latency budget builds this pipeline and times each stage.
• Design: a voice ordering assistant adds an action, an order, to the same turn budget.
• Design: an AI customer support platform puts voice next to chat and email.

Voice gets Amara’s words to the model. Next, a customer sends a photo of a broken lid instead, and asks which part to order.

Common mistakes

Check your understanding

Summary