IllustrationTime to first audio is the sum of seven stages: mic buffering, end-of-turn detection, uplink, model time to first audio, speech synthesis, downlink buffering and the output path. End-of-turn silence is the largest published item, about 800 ms by default on Gemini Live. Set your own target, measure the other stages on your board, and tune silence first.
A voice assistant feels slow long before the model is slow. Most of the wait sits in places a firmware team controls: how long the device waits to decide you stopped talking, how deep the audio buffers are, and how fast playback stops when you interrupt. This guide turns the official realtime API documentation into a budget template you can fill in with numbers measured on your own board.
We publish no measured latency results here. Where a number comes from a vendor page, it carries a link and the date we read it. Everything else is a variable for you to replace.
What stages make up the budget?
Time to first audio, from the moment you stop speaking to the moment the speaker makes sound, is the sum of these stages:
T_first_audio = T_mic_buffer + T_end_of_turn + T_uplink + T_model_first_audio + T_tts_first_chunk + T_downlink_jitter + T_output_buffer
Set T_target as the number you want to stay under. We do not publish a universal target, because it depends on your product and users. Pick yours first, then see what the end-of-turn stage leaves for everything else.
What do the official docs fix, and what is left to measure?
Only a few inputs are published by the vendors. The table shows them in the order the audio travels. Rows whose source is your own firmware, network or test have no official figure in the pages we reviewed, so you measure them on your board.
| Stage | Input | Source |
|---|---|---|
| Mic capture and frame buffer | Frame size you choose in firmware, plus any DMA double buffer | Your firmware Measure on your board |
| End-of-turn detection | Server silence_duration_ms of about 800 ms by default; 500 to 800 ms recommended | Google AI docs as of October 31, 2026 |
| End-of-turn detection, device-side | Silence threshold of at least 500 ms for a client voice activity detector | Google AI docs as of October 31, 2026 |
| Uplink and network round trip | Your Wi-Fi or LTE path to your relay or vendor endpoint | Your network Measure on your board |
| Model time to first audio | Depends on model, region and load. No official figure in the pages we reviewed | Vendor and your test Measure on your board |
| Speech synthesis first chunk | Zero if the model returns audio directly. Otherwise your TTS first-chunk time | Architecture choice Measure on your board |
| Downlink jitter buffer | Buffer depth you choose to ride out network jitter | Your firmware Measure on your board |
| Codec, I2S and amplifier output buffer | Buffer depth in your audio driver and codec; sample rate 24 kHz out for Gemini Live | Google AI docs as of October 31, 2026 |
Two published values anchor the table. Google documents a server default of about 800 ms for silence before the end of a turn, recommends 500 to 800 ms, and warns that values near 100 to 200 ms split single utterances (Live API guide, as of October 31, 2026). For a device that runs its own detector, the same vendor advises a threshold of at least 500 ms (capabilities page, as of October 31, 2026).
Where do the milliseconds go?
Substitute your own target into the formula. If your T_target is close to 800 ms, the default end-of-turn silence alone would consume all of it, before any network or model time. That is the main point of the budget. Silence tuning decides how much room the other six stages get.
Levers the vendors document
- Silence duration. On OpenAI Realtime, silence_duration_ms sets how much silence ends a turn, and shorter values detect turns more quickly (turn detection guide, as of October 31, 2026). On Gemini Live, the range is 500 to 800 ms with an 800 ms server default.
- Server VAD versus semantic VAD. OpenAI offers server_vad, which splits on silence, and semantic_vad, which uses a classifier on the words spoken. Semantic mode has an eagerness setting: auto is the default and equals medium, high chunks audio as soon as possible, and low lets the user take more time.
- Local detection plus manual turn control. The device decides the turn ended and tells the service. You gain control and own the false-cut rate.
We do not quote an OpenAI default for silence duration or threshold, because the page we reviewed does not state one.
How does the audio get to a device that is not a browser?
OpenAI documents that a Realtime session connects over WebRTC in the browser or WebSocket on the server (Realtime guide, as of October 31, 2026). A microcontroller or embedded Linux device sits in neither group. Plan an architecture decision early: a relay you operate that holds the key and speaks to the vendor, or short-lived credentials issued to the device. Each adds a hop, so it belongs in the uplink and downlink rows of your budget. If your product also has data residency rules, see voice wearables and cloud location.
Pitfalls you can predict from the documentation
These follow from the vendor facts above. They are engineering reasoning, not reports of incidents at any company.
- Interruption without a flush. Google documents that when VAD detects an interruption, the ongoing generation is canceled and discarded (capabilities page, as of October 31, 2026). The service stops sending audio, but anything already in your jitter buffer, DMA ring and codec FIFO keeps playing. Playback continues after the user has barged in unless firmware clears those buffers at the same moment.
- Sample rate mismatch. Input is 16 kHz and output is 24 kHz on Gemini Live. A codec clocked for one rate and a resampler for the other adds a stage and its buffer to your budget.
- Aggressive silence tuning. Dropping the threshold to buy time can fragment one utterance into several turns, which the vendor documentation warns about. Each false cut triggers a model response the user did not want.
- Counting only network time. Teams that benchmark the vendor from a laptop miss the buffering stages that exist only on the device.
- Always-on capture. Any wake or turn logic must make capture state obvious to the user. A device that streams audio before the user intends it to is a privacy and legal problem, not a latency feature.
What should you measure on your own board?
This is a method, not a set of results. Run each measurement on your hardware, record the date and region, and use the results to check the budget above.
| Measure | How | How to set the pass line |
|---|---|---|
| Mic to DMA frame latency | Feed a known click or tone at the microphone, capture the GPIO toggle when the first frame reaches the processing task, and compare on a logic analyzer or oscilloscope. | Subtract it from T_target first. Set the line as a fraction of your own target and write it down before you measure. |
| Local VAD hangover | Play a recorded phrase with a known end. Log the time from the last audible sample to your end-of-speech flag. Repeat across quiet and noisy rooms. | Stay at or above the 500 ms vendor floor if you use it for manual turns. Also count false cuts across your phrase set and set a maximum rate. |
| Uplink round trip | Timestamp a ping-style message through your relay or vendor endpoint from the device, on the networks your users have, at different times of day. | Use a high percentile, not an average. Set it from your target and the other stage values. |
| Time to first audio per vendor | Send identical recorded prompts to each service from the same device and network, same day, and log from end of your speech to first audio sample received. Record model name, region and date. | Compare vendors only from same-day runs. Preview models change, so re-run before any decision. |
| Output buffer | Log the time from the first received audio sample to sound at the speaker, using a measurement microphone or a loopback tap. | Derive it from buffer depth and sample rate, then confirm by measurement. Fail the build if measured and calculated values disagree. |
| Barge-in cancel time | Speak over playback. Log from the start of your speech to silence at the speaker, with and without the buffer flush. | Set a maximum before you test, and require the flush path to meet it every build. |
Run enough repetitions to report a spread, and keep the raw logs with the firmware revision that produced them.
Questions to ask before you commit to a design
- What is our T_target, and who signed off on it?
- Which turn-detection mode do we use, and what false-cut rate can we accept?
- Does the device run its own voice activity detection, and who owns tuning it per environment?
- What relay or credential scheme keeps the vendor key off the device, and how many milliseconds does it add?
- Which sample rates run end to end, and where does resampling happen?
- What is the buffer depth at every stage, and does barge-in flush each of them?
- Which vendor model names and defaults are preview-grade, and who re-checks them before each release?
- If a supplier quotes a latency figure, which board, network, region and date was it measured on?
Questions about voice latency
What is the biggest single item in a voice LLM latency budget?
End-of-turn silence. Google documents a server default of about 800 ms for its Live API automatic voice activity detection, so the device waits that long before the model even starts. Shortening it is the largest lever, and the same page warns that very short values split one utterance into fragments. Gemini Live API guide, as of October 31, 2026.
Should the device run its own voice activity detection?
You can. Google's Live API supports manual turn control, and its documentation advises a client-side end-of-speech threshold of at least 500 ms. Local detection lets you start the uplink and drive the user interface sooner, but you then own the tuning. Live API capabilities, as of October 31, 2026.
What sample rates does the audio path need?
For Google's Live API, input audio is raw little-endian 16-bit PCM, natively 16 kHz, and output is always 24 kHz. Design the codec, I2S clocks and any resampler around those two rates. Check the vendor page for your model before you freeze the audio path. Gemini Live API guide, as of October 31, 2026.
Can the device connect to OpenAI Realtime directly?
OpenAI documents WebRTC for browser clients and WebSocket for servers. A device is neither, so plan for a relay you control or short-lived credentials, and keep the long-lived API key off the device. That last part is our engineering judgment, not text on the page. OpenAI Realtime guide, as of October 31, 2026.
What is the default silence setting for OpenAI turn detection?
We do not quote one. The OpenAI turn detection page describes silence_duration_ms and says shorter values detect turns more quickly, but the page we reviewed does not state a default. Read the current value from your own session configuration. OpenAI turn detection guide.
If you are planning a voice device and want the microphone, audio path and firmware designed together, read about audio and voice hardware development or see how the engineering gates run, then tell us about your device.
Primary sources
- https://ai.google.dev/gemini-api/docs/live-guide (accessed October 31, 2026)
- https://ai.google.dev/gemini-api/docs/live-api/capabilities (accessed October 31, 2026)
- https://developers.openai.com/api/docs/guides/realtime-vad (accessed October 31, 2026)
- https://developers.openai.com/api/docs/guides/realtime (accessed October 31, 2026)
Planning a voice or audio device? Tell us what it needs to do. A Duobay engineer replies within 1 business day. Send a project brief →