Low-Latency Speech Translation Pipelines: A Practical Guide
Real-time speech translation only works when the round-trip stays under a second. Above that, conversations stall, speakers talk over each other, and the illusion of a shared language collapses. This guide walks through the architecture and trade-offs behind sub-second multilingual voice — the same pipeline that powers SentiVue's live events and telecom products.
Why latency is the whole product
For batch translation, quality is the metric. For live conversation, latency is quality. A perfectly translated sentence delivered 2.5 seconds late is worse than a slightly rougher one delivered in 700 ms. Humans tolerate roughly 800 ms of end-to-end delay before turn-taking breaks down; beyond that, the exchange stops feeling like a dialogue and starts feeling like a walkie-talkie.
Every stage in the pipeline burns budget:
| Stage | Typical cost | Budget target |
|---|---|---|
| Capture + jitter buffer | 40–120 ms | ≤ 80 ms |
| Streaming ASR (partial → stable) | 200–600 ms | ≤ 300 ms |
| Machine translation | 80–250 ms | ≤ 150 ms |
| Streaming TTS (first audio chunk) | 150–500 ms | ≤ 200 ms |
| Network + playout | 60–200 ms | ≤ 100 ms |
Miss the budget in one stage and no amount of optimization downstream can recover it.
The pipeline, stage by stage
1. Capture with a tight jitter buffer
Most latency regressions we see in production start at the microphone. A generous jitter buffer smooths audio but adds fixed delay to every utterance. For conversational use, keep the buffer at 20–40 ms and lean on packet-loss concealment instead. Use Opus at 20 ms frames over WebRTC or a low-overhead SIP trunk — not HTTP chunked upload.
2. Streaming ASR with stable partials
Batch ASR waits for a full utterance before returning text. Streaming ASR emits partial hypotheses every 100–300 ms and revises them as more audio arrives. The trick is deciding when a partial is stable enough to send downstream. Wait too long and you burn budget; commit too early and you translate a word the speaker is about to correct.
A practical heuristic: commit a token once it has survived two consecutive partials unchanged, or after 250 ms of silence following it — whichever comes first. This is the single biggest lever on perceived latency.
3. Incremental machine translation
Modern MT models can translate incrementally, but naive streaming produces jittery output because word order differs across languages. Two techniques that hold up in production:
- Prefix-constrained decoding. Re-translate the growing prefix on each ASR revision, but only emit tokens that agree with the previous translation. This trades a small quality hit for stable output.
- Wait-k policies. Delay translation by k source tokens. Higher k means better quality but more latency. For most language pairs, k = 3 is the sweet spot.
For dialect-sensitive pairs — Gulf Arabic ↔ English, Brazilian Portuguese ↔ European Portuguese — pick a model fine-tuned on the target dialect. Generic multilingual models regress badly on regional forms and add repair cost downstream.
4. Streaming TTS with first-chunk optimization
The metric that matters is time-to-first-audio, not total synthesis time. A TTS engine that produces the full waveform in 400 ms is worse than one that emits the first 200 ms of audio in 180 ms and streams the rest. Prefer neural vocoders with incremental synthesis (WaveRNN, streaming HiFi-GAN, or vendor equivalents) and pre-warm the voice on session start so the first request doesn't pay a cold-start cost.
5. Transport and playout
Return audio over the same WebRTC or SIP channel that carried the source — a separate HTTP fetch adds a full RTT you cannot afford. Play out as soon as the first chunk arrives; do not wait for the utterance to complete.
Common failure modes
- Coldstarts on the ASR/TTS models. Keep at least one warm replica per language pair. Autoscaling on request count alone will pin the first speaker of every hour with a 2 s penalty.
- Head-of-line blocking on a single WebSocket. Multiplex ASR partials and TTS chunks on separate channels, or use QUIC.
- Serial pipelines. ASR → MT → TTS as a strict chain wastes budget. Overlap: start MT on the committed prefix while ASR is still processing the tail; start TTS on the first translated tokens.
- Language detection on the hot path. Detect language once per session, not per utterance. Redetection adds 100–200 ms to every turn.
What we ship at SentiVue
Our voice stack runs the full pipeline in-region across our telecom edge, with warm model pools per language pair and streaming everywhere between capture and playout. Median round-trip on our production events traffic sits between 650 and 800 ms across the twenty language pairs we support — inside the conversational threshold, even on international routes.
If you're building a live translation product, the shortest path is: measure each stage independently, budget aggressively, and refuse to ship anything that puts the round-trip over one second. The user experience cliff is real, and it is exactly where the market opens up.
