Cutting First-Token Latency in Production Voice AI

The budget: 800ms is the number, and every component has a share of it.

Eight hundred milliseconds is the threshold at which a pause in a telephone conversation stops feeling like processing and starts feeling like a dropped call. The carrier standard for PSTN hold tolerance is somewhere between 600ms and 1,200ms depending on who you ask, but in practice, anything past 800ms starts costing you CSAT points. That is the number we build to.

The budget is not a single component. It is a sum: STT transcription, intent routing, model inference, and TTS synthesis. Each one owns a slice. If any slice runs long, the whole number blows past the threshold. Discipline on each component is the only way to hold the aggregate.

Speech-to-text: the stage with the most variance and the least obvious knobs.

STT is where most of the latency budget gets spent and where most of the variance lives. Streaming transcription helps (you don't wait for end-of-utterance before routing), but the first-token delay still depends on model size, endpoint detection accuracy, and network path to the inference node.

What we changed

• Switched to endpoint-local inference for telephony workloads. Eliminating the round-trip to a hosted API cut our median STT latency from 180ms to 62ms.
• Tuned VAD aggressiveness for contact-center noise profiles. Default VAD settings assume clean mic input. Contact centers have keyboard clicks, hold music bleed, and open-plan background noise. We retrained the endpoint detector on 400 hours of actual carrier recordings.
• Parallel streaming to the router (the intent classifier receives partial transcripts and starts scoring intents before the utterance is complete). This does not always pay off, but when the utterance is a short lookup (claim number, policy ID), it saves 60-80ms.

The most expensive line in the STT budget is the one you can't see: the round-trip to a hosted API you didn't know you were paying for because the vendor counts latency from their server, not from your IVR.Internal latency audit · Q1 2026

Intent routing: small model, fast decision, almost always right.

The router's job is narrow: given a partial or complete transcript, decide which workflow to hand off to, in under 20ms. We run a fine-tuned classifier at ~110M parameters for this, not a frontier model. Frontier models are accurate, but they're 300-500ms by themselves even with streaming. A smaller, domain-specific model that's 98.4% accurate on your actual call distribution beats a general-purpose model that's 99.1% accurate everywhere.

Model inference: where you spend the budget you saved everywhere else.

The LLM stage is where you spend the budget you bought by being disciplined in the earlier three. After routing, the model has a constrained task: generate a response within the conversation context, tool-call the relevant system if needed (claims lookup, schedule read), and return a response token stream. We target first token under 280ms on p50.

Two decisions matter most here: the context window you pass (smaller is faster) and whether tool calls run in parallel. If a workflow requires both a CRM lookup and a claims-system read, sequential calls will blow the budget. Every workflow we ship has a concurrency map that specifies which tool calls can fire simultaneously.

TTS: the last stage, and the one callers notice most.

TTS is where the model's token stream becomes an audio signal the caller hears. We stream TTS; the first audio chunk goes to the caller as soon as the first sentence boundary is available in the token stream. This means a caller starts hearing the response before the model has finished generating the full answer.

The voice is a product decision. We test three to five voices per deployment, with real call recordings and real caller feedback. CSAT variance across voices on the same workflow can be 0.4-0.6 points. That is not a small number. Pick the voice the way you'd pick the hold music: deliberately, with data.

The three changes that moved the needle from 1.1s to 612ms.

After a quarter of production telemetry, three changes account for most of the improvement. They are not the three most technically interesting changes. They are the three most operationally boring ones that nobody does because they require discipline across teams, not a new model.

• Local STT inference. Cut the network round-trip. 118ms median reduction.
• Parallel tool calls. Where two system reads were sequential, we made them concurrent. 80ms median reduction on workflows that hit both.
• Context window discipline. We audited every prompt template and cut the median context size by 38%. Smaller context means faster first token. 60ms median reduction.