Summary
-
Native-audio models have changed the voice AI inference stack. Instead of requiring an application to explicitly chain speech recognition, a language model and text-to-speech, models such as Gemini Live can consume and generate continuous audio directly.
-
Real-time translation has moved forward as well. Gemini 3.5 Live Translate is now a dedicated audio-to-audio translation model, with an explicit translation configuration rather than translation being controlled only through a conversational system prompt.
-
What has not changed is the boundary between the model and the telephone network. A model session processes streams and maintains inference state. A call platform manages SIP signalling, participants, call legs, conferences, media routing and transfers.
-
This distinction becomes critical during AI-to-human escalation. If multilingual behaviour or translation exists only inside the AI worker that is leaving the call, there is nothing inherent in the model session that causes that capability to move to the human participant.
-
Translation that must survive a handoff therefore needs an independent lifecycle at the call or media layer. The model can provide the translation engine, but the surrounding system has to keep that engine correctly attached to the changing call topology.
Native-audio models have removed a lot of complexity from voice AI. They have not removed the call architecture around them.
That distinction is easy to miss because, during an AI-only conversation, the boundaries can look almost invisible. Audio arrives from the caller, the model understands it, and audio comes back. Whether the application is using an explicit STT → LLM → TTS pipeline or a single native-audio session may not be obvious from the caller’s perspective.
The difference becomes much more important when the AI is no longer the endpoint.
A customer asks to speak to a person. The system finds an available agent. The human joins the call. The AI leaves.
If the caller and the human do not speak the same language, translation now has to continue across a participant transition that the inference model itself does not own.
That is the translation gap.
Native audio changed the inference pipeline, not the call boundary
The conventional voice AI architecture is straightforward to describe. Incoming audio is decoded and passed to automatic speech recognition. The resulting text is sent to a language model. The model produces a textual response, and a text-to-speech engine synthesises audio for playback to the caller.
Because those components are distinct, the application can reason about them independently. ASR can use one provider, reasoning another, and synthesis a third. Language selection, speaker selection and audio generation are exposed at explicit boundaries in the pipeline.
Native-audio systems move much of that work inside the model. Google’s current Live API, for example, accepts continuous audio over a stateful WebSocket and produces audio directly. Its native-audio models also expose transcription, voice activity detection and multilingual behaviour without forcing the application to reconstruct the old three-model pipeline.
That does not mean every surrounding concern disappears into one model. The model still receives audio through an API connection. The telephone call still arrives through a carrier or SIP provider. Media still has to move between those two systems. Call signalling, participant membership, transfer logic, hold state and media routing remain outside the inference session.
This matters because native audio has changed where speech intelligence is performed. It has not turned a WebSocket inference session into a SIP call.
There is also an important difference between a multilingual agent and a translator
This is the first architectural distinction that has to be made correctly.
A native-audio AI agent can itself operate in the caller’s language. Gemini Live, for example, supports multilingual speech interaction and can switch languages within a conversation. In that architecture there may be no separate “translation service” at all. The AI receives the caller’s speech and directly produces customer-facing speech in the appropriate language.
That works well while the AI is the participant speaking to the customer.
A dedicated translation model is different. Google now explicitly separates Live Agent from Live Translation in the Gemini API. The Live Agent behaves as an assistant: it reasons, follows instructions, can use tools and participates in a turn-based interaction. Gemini 3.5 Live Translate behaves as an interpreter: it continuously receives audio and produces translated audio, supports audio input only, and does not expose tools or system instructions.
That distinction changes the handoff problem.
If an AI agent has simply been speaking Portuguese directly to a Portuguese caller, there is no independent Portuguese-to-English interpreter waiting somewhere inside the call. When an English-speaking human replaces that AI, a translation path now has to exist between two human endpoints.
If the system was already using a dedicated translator in front of the AI, the problem is different but related. The translator exists, but its lifecycle and media routes have to remain valid after the AI participant is removed.
In both cases, the central question is the same: what owns multilingual continuity when the endpoint changes?
The answer cannot simply be “the model session”, because the model session and the call session are not the same state machine.
A model session and a telephony session have different lifecycles
Gemini’s Live API describes a session as a continuous stream of audio or text between the application and the model. The developer establishes the session, sends realtime input, receives model output, manages context and can use session-resumption mechanisms to reconnect after certain interruptions. Google documents this lifecycle explicitly, including session limits, resumption handles, context management and WebSocket reconnection.
Telephony uses a different set of primitives.
A SIP environment deals with callers and callees, SIP dialogs, INVITEs, transfers, media negotiation and call termination. Media may flow using RTP or SRTP. A higher-level media system may represent the same things as rooms, participants and tracks. LiveKit, for example, represents a telephone caller as a SIP participant inside a room while keeping SIP signalling and realtime media transport separate from the AI agent running in that room.
The important point is not which telephony stack is used. The important point is that these objects exist independently from the inference connection.
A call can remain active after a model WebSocket disconnects.
A model session can be resumed while the underlying telephone leg stays connected.
A human can be added to a conference without becoming a new role inside the AI model’s session.
A SIP transfer can establish a new destination leg while the original model knows nothing about that signalling event unless the application tells it.
Once those lifecycles are separated, the engineering problem becomes much clearer.
A handoff is a change to the media graph
It is tempting to describe escalation as “switching from the AI to a human”. That is useful product language, but technically it hides most of what has to happen.
A production handoff changes the topology of a live realtime system.
Before escalation, a simplified call might look like:
Caller → telephony media → AI worker → realtime model
The exact arrangement varies. A media server may sit between the carrier and the AI worker. Translation may be inside the model, in a separate realtime service, or absent because the AI is already speaking the caller’s language.
Now suppose the caller needs a human.
If the implementation simply removes the AI worker and transfers the telephone leg elsewhere, every resource whose lifecycle was tied to that worker can disappear with it. That can include the model WebSocket, local audio buffers, transcription state and, if translation was instantiated inside the same process, the translation connection itself.
The translation model has not failed in that situation. The application has destroyed the path through which translation was happening.
A more resilient topology keeps a call-level media anchor alive while participants change around it. During a warm handoff there can briefly be three relevant participants or services attached to the same media environment: the caller, the AI and the human. Translation remains attached to that environment independently of the AI. Once the human is ready, the AI can be detached without tearing down the media path used by the other two participants.
This architecture is not specific to SentiVue. Google’s own Agent Assist live audio-to-audio translation product uses conference mode for essentially this reason. Its translation service joins a multiparty call, receives the conversation audio, processes the translation and returns translated speech to the conversation rather than being implemented as a property of a conversational AI agent. Google’s current implementation is a distinct contact-centre product with a narrower language surface than the underlying Gemini 3.5 Live Translate model, which itself illustrates the difference between model capability and telephony product architecture.
Cold transfer and warm handoff have very different consequences
The word “transfer” is also too broad to describe the engineering behaviour.
A cold SIP transfer can move the caller out of the media environment that was hosting the AI. RFC 5589 describes SIP transfer using mechanisms including REFER and Replaces. When REFER causes the transferee to establish a session with the transfer target, the media negotiated for that new session is independent of the media that existed between the original parties.
That is exactly what platforms such as LiveKit expose in practice. Its cold-transfer flow sends SIP REFER through the trunk, connects the caller to the destination and closes the caller’s existing LiveKit session once the transfer completes.
For a normal call, that may be precisely what you want.
For a translated call, it can be a problem.
If the translation service was receiving audio from the old room or media stream, transferring the caller out of that environment means the translator no longer has the audio it was translating. There is no SIP primitive that automatically takes an arbitrary AI translation WebSocket and attaches it to the new call.
You have two choices. Recreate the translation topology at the destination, including the relevant language and state, or keep the caller anchored in a media environment that survives the handoff.
A warm handoff makes the second pattern possible. The system can keep the original caller leg alive, dial the human as another participant, perform a private consultation if required, join the human to the customer’s media environment, and only then remove the AI. LiveKit’s current warm-transfer implementation is instructive because it explicitly models separate caller and consultation sessions, conversation history, participant movement, failure recovery and the final removal of the AI agents.
For translation continuity, that is the important property of a warm handoff: the system can change who is speaking without first destroying the media environment on which translation depends.
The persistent object should be the call, not the AI worker
This is the architectural principle underneath the whole problem.
In a tightly coupled implementation, the AI worker owns everything. It owns the telephony stream, the model session, the translation connection and the application state. When that process exits, everything associated with it exits as well.
That can be fine for a simple AI-only call.
It is fragile once you introduce escalation, conferences, reconnects or multiple realtime services.
A more durable implementation treats the call or media controller as the persistent object. The AI agent and translator are services attached to that object.
The distinction is subtle but important:
Coupled architecture: AI worker lifetime = media lifetime = translation lifetime
Decoupled architecture: call lifetime > AI worker lifetime, with translation and agent services attached or detached independently.
The media anchor might be a conference server, a room, a media bridge, a back-to-back SIP user agent, or a provider-specific bidirectional media stream. There is no single required implementation. What matters is that the component controlling call continuity does not disappear just because one inference participant has finished its job.
This also creates a cleaner failure boundary. If the AI model connection has to reconnect, the telephone call can remain established. If the human transfer fails, the AI can remain available rather than having already destroyed its own media path. If the translation service has a temporary failure, the system can retry or degrade explicitly without automatically terminating the underlying call.
Those are infrastructure properties, not model capabilities.
The audio boundary creates another set of problems
Even when the participant topology is correct, the media path still has to be engineered properly.
Consider a common integration using Twilio Media Streams and Gemini Live Translation. Twilio sends call audio as mono audio/x-mulaw at 8 kHz. Audio returned to a bidirectional Twilio Media Stream also has to be μ-law at 8 kHz. Gemini’s Live Translation interface specifies raw little-endian 16-bit PCM at 16 kHz for input and produces raw 16-bit PCM at 24 kHz for output.
The production media path therefore has conversion work to do. Conceptually, the caller-to-model direction may be:
PSTN μ-law/8 kHz → decode → resample → PCM/16 kHz → translation model
and the return path:
PCM/24 kHz → resample → μ-law/8 kHz → PSTN
The exact implementation will depend on the carrier, media server and model provider. Google notes that its general Live API can resample input audio, but the Live Translation guide still specifies 16 kHz PCM as the expected input format, so normalising at the application boundary gives the system an explicit and testable media contract.
This is more than format conversion. Every component can affect latency.
For a realtime translated call, end-to-end delay is roughly the sum of capture and chunking time, ingress network latency, decoding and resampling, model translation latency, egress network latency, output conversion and the final telephony playout buffer.
The model is only one term in that budget.
Google recommends 100 ms input chunks for Gemini Live Translation. Twilio independently buffers outbound media sent to a bidirectional stream and plays those messages in order. The application therefore has to control how much audio it accumulates before inference, how quickly it returns generated audio, and how much unplayed audio it allows to queue downstream.
A model can have excellent measured translation latency and still produce a poor call if the integration adds unnecessary buffers around it.
Handoff also exposes the problem of stale audio
Realtime voice systems rarely have only one audio buffer.
The model provider may have pending generation. The application may have decoded or transcoded audio waiting to be sent. The telephony provider may already have generated audio queued for playback.
That matters during interruption and handoff.
Gemini’s Live API tells the client when an interaction has been interrupted and requires the application to handle already-buffered playback appropriately. Google’s troubleshooting guidance specifically identifies failure to flush the playback buffer as a cause of the model apparently continuing to speak after interruption.
Twilio has a separate mechanism at the telephony layer. Media sent into a bidirectional stream is buffered until playback, and a clear message empties audio that has not yet been played. Twilio’s mark mechanism can also tell the application when a particular section of generated audio has actually finished playing to the caller.
That gives an engineering team two different behaviours to handle during escalation.
If the AI says, “I’ll connect you to an agent now”, and that sentence needs to finish before the human enters, the application can track the completion of the final media segment before changing state.
If the human is already entering and the AI’s queued output must stop immediately, the system should cancel generation and clear downstream audio rather than waiting for the remaining queue to drain.
Without that coordination, the handoff can be logically complete while stale AI audio is still physically travelling towards the caller.
That is how systems end up with the AI talking over the human agent even though the application believes the AI has already left.
A translation bridge also has to avoid translating itself
Multiparty translation creates another media-routing problem: the output of the translator must not unintentionally become new input to the translator.
Suppose the caller speaks Arabic and the human agent speaks English. Audio from the Arabic caller needs to reach the translation engine and then the English-speaking agent. Audio from the English-speaking agent needs the corresponding path back towards the caller.
If translated output is simply injected into a conference mix that is then fed back to the same translation input without participant or track filtering, the system can create feedback, duplicate speech, or ask the model to process its own generated output.
The cleanest implementation depends on the telephony platform. Separate participant tracks make directional routing explicit. A conference service can instead perform speaker attribution and ensure that generated translation is routed to the correct endpoint without becoming another source stream. Google’s Agent Assist conference implementation explicitly processes multi-participant audio and routes translated output back into the conversation, while platforms such as LiveKit expose participants and audio tracks independently at the media layer.
This also explains why speaker separation is not just an analytics feature in translated calls. It is part of the routing logic.
The issue becomes harder during overlapping speech. Google’s own model card for Gemini 3.5 Live Translate notes known limitations around rapid multi-speaker conversations, including voice assignment instability, as well as challenges with heavy accents, similar languages and fast language switching.
A well-designed media topology cannot remove model limitations, but it can avoid introducing additional ambiguity before the audio reaches the model.
Language state has to survive the handoff as well
Translation continuity is not only an audio-routing problem.
The system also needs to know what each participant is supposed to hear.
Gemini Live Translation exposes a targetLanguageCode and an echoTargetLanguage setting. The latter determines what happens when incoming speech is already in the target language: the model can echo that speech or stay silent. The service can automatically detect the input language, but its own documentation notes edge cases around accents, similar languages and rapid switching.
That is sufficient for a translation session.
A contact-centre call has additional state.
The customer might be speaking Portuguese. The AI might have been operating directly in Portuguese. The human agent might speak English. The application now needs to know that the customer-facing translation direction is English → Portuguese and the agent-facing direction is Portuguese → English.
If a human agent with Portuguese capability answers instead, the correct behaviour may be to remove translation entirely.
If the next human speaks Spanish rather than English, the translation configuration changes again.
That language mapping belongs in application or call state, not only inside a model connection. Otherwise the system has no canonical record from which to rebuild the translator when participants change or a realtime connection has to be recreated.
“Context” also needs to be decomposed
Voice AI platforms often say that a human agent receives “full context” during handoff.
Technically, that phrase is too vague to be useful.
There are several different kinds of state that may need to survive.
Model context is the information currently represented inside the active inference session. It can include previous turns and model-side conversation state.
Conversation context is material that the application can persist independently, such as transcripts, detected language, customer intent, entities and a structured handoff summary.
Business state includes information obtained from tools or backend systems: authentication results, bookings, account records, order status, support tickets and any actions that have already been performed.
Call state describes the live communications session itself: participant identities, call or room identifiers, hold state, media routes and whether a transfer is pending, merged or complete.
These states should not be treated as interchangeable.
Gemini’s session-resumption mechanism can restore model-side context after a supported disconnection, but it does not transfer a telephone participant or make hidden model context visible to a human agent. LiveKit’s warm-transfer workflow illustrates the complementary pattern: conversation history is explicitly passed into the transfer workflow, while participant and room state are managed separately by the telephony application.
Anything the human must rely on should therefore be materialised outside the departing AI session.
Persist the transcript that matters.
Persist the language mapping.
Persist relevant tool results.
Persist authentication and workflow state where the underlying business process permits it.
Generate a concise handoff summary if the agent needs one.
Do not assume that because the AI “knows” something, the next participant automatically does.
A production handoff should be a state machine
Once a handoff touches multiple realtime systems, implementing it as a single transfer() action creates unnecessary race conditions.
A more reliable implementation gives the transition explicit states.
The names are application-specific, but the workflow typically needs to distinguish an AI-owned call, an escalation request, a human connection attempt, a consultation or hold phase, a merged state, AI detachment and a final human-owned state. There also needs to be a defined failure path if the human does not answer or the merge fails. LiveKit’s current warm-transfer implementation follows this type of staged orchestration, including caller hold, a consultation room, context transfer, participant movement and recovery if the target cannot take the call.
For multilingual calls, translation becomes another participant in that state machine.
It should not transition to “stopped” simply because the AI transitions to “detached”.
During the merge, the media router may need to change which streams are translated and where output is delivered.
During AI detachment, the system needs to fence off late model events and prevent old audio from entering the now human-owned call.
During transfer failure, the existing translation and AI paths may need to remain available so that the caller can continue rather than being stranded between two sessions.
This is the kind of logic that does not appear in a model API benchmark, but it determines whether the system behaves correctly under real call conditions.
Failure domains should be independent
There is a simple way to test whether the architecture has been coupled at the wrong level: remove one component and see what else dies with it.
If the AI model’s WebSocket disconnects, does the telephone call terminate?
If the translator reconnects, does the human leg have to be recreated?
If the human transfer target fails to answer, has the original AI already been destroyed?
If the model session reaches a connection limit, can it reconnect while the caller remains on the line?
If one of those failures automatically cascades into unrelated components, the system has made one session responsible for too much.
Google’s Live API provides session-resumption mechanisms precisely because realtime model connections have their own lifecycle and can need reconnection. SIP and media platforms separately maintain call participants and call state. Those independent mechanisms are a useful reminder of the correct architectural boundary: model continuity and call continuity are related, but they are not the same problem.
For carrier-grade voice systems, that separation matters more than the choice of model.
Models will change.
The call still has to stay up.
What this means for SentiVue Talk
This is the layer SentiVue Talk is designed around.
The important architectural decision is not to treat the AI agent as the owner of the call. The call and its media lifecycle remain the persistent object, while AI, translation and human participants can change around it.
That changes what a handoff means.
Instead of shutting down one multilingual session and creating an unrelated human call, the system can preserve the media environment required for translation while changing who owns the conversation. The human can receive application-level context that has been persisted outside the model, and the translation path can continue without depending on the departing AI session.
This does not make the underlying translation model unimportant. Translation quality, latency, language coverage, accent handling and speech naturalness still depend heavily on the model being used. Google’s progress with Gemini 3.5 Live Translate is evidence of how quickly that layer is improving. The model now supports more than 70 languages, continuous speech translation and native audio output, although Google also documents limitations around multi-speaker audio, accents and language detection.
The point is that the model should remain a replaceable part of the system.
A production voice platform should be able to adopt a better translation model without redesigning how calls are transferred, how participants are represented, how audio is routed, or where business state lives.
That is the architectural difference between adding translation to an AI agent and building translation into a call platform.
What to ask a voice AI vendor
If multilingual escalation matters to your use case, asking whether a vendor “supports translation” is no longer a useful technical test.
Ask what happens to the actual media when the AI leaves.
Ask whether the caller remains anchored in the same room, conference or media bridge, or whether the platform performs a cold transfer that creates a new media session.
Ask whether translation has an independent lifecycle or runs inside the AI worker.
Ask how the system handles the two translation directions when the caller and human agent use different languages.
Ask whether the platform works from separate participant tracks or a mixed stream, and how it prevents generated translation from feeding back into its own input.
Ask how codec conversion and resampling are performed between the telephony network and the realtime model.
Ask what is done with queued model and telephony audio when the handoff occurs.
Ask where language state, transcripts, tool results and authentication state live after the model session is gone.
Ask what happens if the human does not answer.
And ask whether the caller stays connected if the model or translator has to reconnect.
Those answers describe the system you are actually buying.
The model name does not.
Frequently Asked Questions
Can native-audio models translate phone calls in real time?
Yes. Current native-audio models can provide low-latency speech-to-speech translation. Gemini 3.5 Live Translate, for example, accepts continuous audio and produces translated audio and transcripts across more than 70 supported languages. It is exposed as a dedicated translation model rather than only as multilingual behaviour inside a conversational agent.
The engineering issue in a contact-centre handoff is not whether the model can translate. It is whether the translation service remains correctly connected to the media after the participant topology changes.
What is the difference between a multilingual AI agent and a translation service?
A multilingual AI agent is the conversational endpoint. It understands the caller and generates its own response in the caller’s language.
A translation service is an interpreter between other participants. Google’s current API makes that distinction explicit: its Live Agent supports agent behaviours such as instructions and tools, while Live Translation is an audio-only continuous translation pipeline without those agent capabilities.
That difference matters when the AI is replaced by a human, because multilingual behaviour that belonged to the AI does not automatically become a translator for the two people who remain.
Why can translation disappear during an AI-to-human handoff?
Translation disappears when its media path or process has been coupled to the AI component that is being removed.
For example, if a cold transfer sends the caller out of the original conference or room, any translation service attached to that environment loses its original media source unless the application explicitly recreates or preserves the translation path. SIP REFER transfer behaviour and current telephony implementations demonstrate that the destination call’s media is not simply the old media session with a different participant name.
Is a warm transfer required?
Not necessarily. A cold transfer can still support translation if the platform explicitly creates the required translation path on the destination side.
A warm or conference-based handoff is often easier to reason about because the caller can remain attached to a persistent media environment while the human joins and the AI leaves. This allows the translation service to remain in place during the transition. Google’s live Agent Assist translation architecture and LiveKit’s warm-transfer workflow both use persistent multiparty media constructs rather than treating the handoff as a simple change to an AI prompt.
Doesn’t Gemini session resumption solve continuity?
It solves a different form of continuity.
Session resumption can restore a Gemini Live session after a supported WebSocket interruption and preserve model-side conversational context. It does not preserve SIP signalling, connect a human agent, control a conference, or route translated media between telephone participants. Those responsibilities remain in the surrounding application and telephony layer.
What does a translation service need to preserve through handoff?
At minimum, the system needs a valid media path to both participants and an explicit mapping of which language each participant should receive.
In a production implementation it will usually also need participant identity, call or room state, language configuration, transcript or handoff context, and coordinated control of any queued audio. The exact state depends on the application, but it should live somewhere that survives the lifecycle of the AI session that is leaving.
That is the translation gap: not whether today’s models can translate speech, but whether the voice infrastructure can keep translation working when the call changes underneath them.
