Real-time generative AI is a different engineering problem from batch generative AI, and most teams discover this the first time a demo meets a real user typing fast. The model that summarised a document in eight seconds offline now has to start speaking within 300 ms, hold a conversation under flaky network conditions, and cancel itself cleanly when the user interrupts. None of that is a model problem. It is a streaming problem — and the patterns that solve it look very little like the request/response code that worked in the prototype. We work on this layer often, usually after a batch GenAI pipeline has been lifted into an interactive UX and the latency wall has shown up. The fix is rarely “use a smaller model”. It is almost always a redesign of how partial outputs flow from the model to the user, how the consumer applies backpressure, and how the platform renders incremental audio, text, or pixels. What real-time streaming actually means in a generative system Two patterns get bundled under the same label, and conflating them is the first place architectures go wrong. The first is output streaming: the model produces tokens (or audio frames, or image refinements) incrementally, and the client renders them as they arrive. This is the ChatGPT-style UX. The transport is typically HTTP/2 server-sent events for text, gRPC streaming between services, or WebRTC for audio and video. The serving layer matters: vLLM, TensorRT-LLM, SGLang, and Triton Inference Server are the stacks that handle continuous batching well, and continuous batching is what makes streaming economically viable under load. The second is input streaming: the model continuously ingests audio, video, sensor data, or market data and produces ongoing output. Voice agents are the clearest example. A live captioning system is another. Here the engineering problem is dominated by chunked encoding, voice activity detection, and the bookkeeping required to align partial recognitions with partial generations. The two patterns share a transport layer and a backpressure model. They diverge sharply on how the model is invoked, and any architecture document that treats them as one thing will produce confused trade-offs downstream. Why first-token latency dominates the budget In our experience, the operationally relevant latency measure for real-time GenAI is time-to-first-token, not full-response latency. A 12-second response that starts in 200 ms feels responsive. A 4-second response that starts in 2 seconds feels broken. Users register the gap before the first character, and very little else they wait through feels comparable. The observed ranges we work to as planning heuristics — these are starting budgets, not benchmarks, and they shift per deployment: UX class First-token / first-response target What slips first under load Text chat (responsive) < 500 ms Tokenisation queue depth, prompt-eval time Text chat (instant feel) < 200 ms Network RTT, cold-start KV cache Voice agent round-trip < 800 ms total STT chunking latency, TTS first-audio Voice agent (natural feel) < 500 ms total Same as above plus interrupt handling Live video understanding 100–300 ms perception-to-response Frame batching, encoder throughput These are observed patterns across the deployments we’ve reviewed; treat them as planning bands, not external benchmarks. The honest version is that voice agents almost never hit 500 ms end-to-end on a first attempt — the path through streaming speech-to-text, model inference, and streaming TTS accumulates faster than teams predict. How low-latency TTS trades quality for latency Text-to-speech is where the streaming budget gets tested hardest, because audio cannot be visually buffered the way text can. The user hears the gap. Modern streaming TTS systems — Cartesia, ElevenLabs Streaming, and open systems like Piper — produce the first audio chunk in roughly 100–300 ms after receiving the first text. End-to-end voice models like GPT-4o-voice and Gemini Live collapse the STT-LLM-TTS pipeline into a single model and trade some control for lower turn latency. The trade-off is not abstract: Concatenative or parametric TTS (the pre-neural baseline) can hit very low first-audio latency but sounds robotic and cannot reproduce affect or unfamiliar names well. Neural autoregressive TTS produces near-human prosody but has a non-trivial first-audio cost because the autoregressive prefix has to be primed. Non-autoregressive or diffusion-based TTS parallelises generation and reduces first-audio latency, sometimes at the cost of subtle prosody artefacts on long utterances. End-to-end voice models skip an explicit text intermediate, which helps interruption handling and emotional alignment but makes it harder to inject deterministic content (a specific phone number, a brand name with required pronunciation). There is no single right choice. The decision is governed by whether the application can tolerate occasional pronunciation drift, whether it needs to inject canned segments, and whether the interaction is short and turn-based or open-ended and barge-in-capable. What a streaming LLM looks like architecturally The architectural shift from batched to streaming inference is mostly about state and scheduling. Batched inference treats each request as atomic: prefill the prompt, generate to completion, return the full response. Streaming inference splits the request into a prefill phase and a decode phase, holds the KV cache across the boundary, and emits tokens incrementally to the transport layer. Continuous batching — the technique that put vLLM on the map — lets new requests join the in-flight batch at each decode step, so GPU utilisation stays high even when individual requests stream at different rates. Speculative decoding is the other lever. A small draft model proposes several tokens ahead, the main model verifies them in a single forward pass, and on a verified prefix the system emits multiple tokens per step. We treat this as the cheapest streaming acceleration available today: it reduces effective per-token latency without retraining, and the quality is bit-for-bit identical to the base model when implemented correctly. The thing that breaks in streaming architectures is usually not the model. It is one of four cross-cutting concerns most early implementations underestimate: Cancellation and backpressure. The user stops listening, types a new prompt, or closes the tab. The in-flight generation has to be cancelled cleanly so the KV cache slot is reclaimed and the GPU does not burn cycles on a stream nobody is reading. Tool-call interleaving. The model wants to call a function mid-generation. The stream needs to pause, run the tool, fold the result back into context, and resume — without the client seeing a dead connection. Reconnect-and-resume. Mobile networks drop. A streaming protocol that cannot resume mid-generation will retry the whole prompt, doubling the cost and breaking conversational state. Streaming safety filtering. Content moderation that requires the full output before deciding defeats the streaming UX. Token-level or chunk-level moderation is harder to get right but is the only approach compatible with first-token budgets under 500 ms. Most production streaming bugs we see live in these four areas, not in the model. How the latency budget maps to network, model size, and hardware The latency budget for a real-time GenAI feature decomposes into a small number of additive terms. Writing them down explicitly is the single most useful exercise we do on an audit: Client-to-edge RTT: typically 20–80 ms depending on geography. Co-locating inference in the user’s region is usually worth more than any model-side optimisation when this term dominates. Edge-to-model RTT: 5–40 ms if the model is in the same datacentre; much larger if the edge proxies through a central region. Prefill time: linear in prompt length, roughly milliseconds per token of input on modern GPUs. Long system prompts are a silent first-token cost. Decode time per token: model-size and hardware dependent. A 7B model on an H100 decodes meaningfully faster than a 70B model on the same hardware, but the quality gap can be the deciding factor. Client render time: usually small for text, non-trivial for audio (decoder buffer) and significant for image refinement. If the budget is 500 ms to first token and the prompt is 4,000 tokens long, prefill alone may consume 200–400 ms. Either the prompt has to shrink, the model has to change, or prefix caching has to do real work. There is no fourth option. Where streaming generative AI ships in production today The deployments we see most often, with no commentary on volume: Voice agents in customer support, where the round-trip turn budget governs everything else. Live captioning and translation, where streaming STT feeds a streaming translation model and partial captions are revised in place. Coding assistants, where token-streaming and tool-call interleaving have to coexist with editor latency expectations. Real-time graphics and progressive image generation, where the user sees the image refine rather than waiting for a final render. Live document drafting, where the streaming surface is text but the model is doing structured generation under the hood. Each of these has its own dominant constraint. Voice agents are gated by round-trip latency; live captioning by partial-result revision quality; coding assistants by tool-call latency and context window management; progressive image generation by per-step compute and the perceptual quality of intermediate states. How TechnoLynx approaches real-time GenAI engagements When we engage on a real-time generative AI feature, the first deliverable is almost always a streaming budget broken into the terms above, mapped against a target UX. The second is a feasibility audit — closely related to our work on latency optimisation for production GenAI systems — that tests whether the chosen model, transport, and platform can actually hit the budget under realistic load, not under demo conditions. We pay close attention to the four cross-cutting concerns above because they are where streaming systems quietly degrade. A pipeline that hits its first-token target on a clean network and fails to cancel cleanly when the user interrupts will feel broken in production no matter how good its numbers look in isolation. If you are building a real-time GenAI feature and want to know whether your latency budget is defensible before you ship it, we’d be glad to take a look. Frequently asked questions What does real-time generative AI actually mean — first-token latency, full-response latency, streaming? Real-time generative AI covers two patterns. The first is output streaming: the model emits tokens, audio frames, or image refinements incrementally and the client renders them as they arrive. The second is input streaming: the model continuously ingests audio, video, or sensor data and produces ongoing output. The operationally relevant latency measure in both cases is time-to-first-token (or first-audio, or first-frame), not full-response latency — the user perceives the gap before the first chunk far more sharply than the total duration. How do low-latency TTS systems (Morpheus, Vox, Qwen3-TTS, Piper) trade quality for latency? Concatenative and parametric TTS systems hit very low first-audio latency but sound robotic and handle unfamiliar names poorly. Neural autoregressive TTS produces near-human prosody but pays a first-audio cost from priming the autoregressive prefix. Non-autoregressive and diffusion-based TTS parallelise generation and reduce first-audio latency, sometimes at the cost of prosody artefacts on long utterances. End-to-end voice models collapse the STT-LLM-TTS chain and improve interruption handling but reduce control over deterministic content like specific pronunciations. What is a streaming LLM architecturally, and where does it differ from batched inference? Batched inference treats each request as atomic — prefill, generate to completion, return. Streaming inference splits the request into a prefill phase and a decode phase, holds the KV cache across the boundary, and emits tokens incrementally to the transport layer. Continuous batching lets new requests join the in-flight batch at each decode step, keeping GPU utilisation high. Speculative decoding adds a draft model that proposes tokens ahead of the main model, reducing effective per-token latency without retraining. Where does streaming generative AI ship in production today — live captioning, voice agents, real-time graphics? The common production surfaces are voice agents (gated by round-trip turn latency), live captioning and translation (gated by partial-result revision quality), coding assistants (gated by tool-call latency and context management), progressive image generation (gated by per-step compute and intermediate perceptual quality), and live document drafting. Each surface has a different dominant constraint, which is why a single “real-time GenAI architecture” rarely transfers cleanly between them. How does the latency budget for real-time GenAI map to network, model size, and hardware choices? The budget decomposes into client-to-edge RTT, edge-to-model RTT, prefill time (linear in prompt length), decode time per token (model and hardware dependent), and client render time. Long system prompts are a silent first-token cost. Co-locating inference in the user’s region is usually worth more than any model-side optimisation when network terms dominate. Speculative decoding is the cheapest acceleration available today for decode-bound budgets. What benefits does generative AI for text-to-speech bring over classical concatenative or parametric TTS? Generative neural TTS produces prosody, affect, and unfamiliar-name handling that concatenative and parametric systems cannot reach. It generalises to new voices with limited reference audio, supports multilingual output from a single model, and integrates more cleanly with end-to-end voice agents. The trade-off is higher first-audio latency than the simplest parametric baselines and less deterministic control over specific pronunciations — both manageable with modern streaming TTS stacks and prefix caching. Image credits: Freepik.
