Lowest-Latency Inference APIs for Voice Agents: TTFT

A voice agent functions essentially as a latency budget enclosing a language model, where every processing stage consumes milliseconds that users can directly perceive. Time to First Token (TTFT) measures the interval between dispatching an inference request and receiving the initial token, which IBM defines as the system's transition from idle to active status.

While TTFT dominates text-based chat metrics, voice interactions treat it as merely a single component within a larger sum. Mechanical limitations dictate that text-to-speech models cannot synthesize half a word, requiring complete clauses before generating audio. Consequently, LiveKit highlights Time-to-First-Sentence (TTFS) in its Gemma 4 deployment analysis as the true experiential metric for users.

Stock photo for illustration only, not from the actual event

This dynamic introduces two primary variables: TTFT governs when generation begins, while tokens per second determine how quickly the first sentence completes. A provider excelling in only one metric will fail to deliver a responsive feel. LiveKit's architectural overview estimates turn breakdowns as 100–200ms for STT, 300–500ms for streamed LLM, 100–200ms for TTS, and 50–150ms for WebRTC networking, establishing a practical end-to-end target between 700ms and 1.2 seconds.

Kwindla Hultman Kramer, co-creator of Pipecat, recommends targeting an 800ms median voice-to-voice latency, with 1,500ms serving as an acceptable proof-of-concept threshold. His breakdown allocates roughly 200ms across transport processing, STT endpointing, LLM inference, and TTS. Human conversational baselines identified by Daily show typical responses around 500ms, with unnatural pauses emerging beyond 800ms.