Latency Optimization › Streaming Responses
Language AI HandbookAll chapters Latency Optimization › Streaming ResponsesOne section · Sections in this chapter6 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Streaming is the technique of sending generated tokens to the client as they are produced, rather than buffering the entire response and sending it all at once. For users, the experience difference is dramatic: instead of waiting 20 seconds for a long response and then seeing it all at once, they see words appearing progressively, making the system feel much more responsive.
The perception effect of streaming is well-documented in user research on web applications. Users consistently rate streaming interfaces as faster even when the total completion time is identical to the non-streaming version. This happens because streaming provides visual feedback that something is happening, reducing anxiety about whether the system has accepted the request, and because reading can begin before generation is complete for long responses. Streaming is therefore one of the highest-impact latency optimizations available, and it costs essentially nothing to implement in most serving frameworks.
How Streaming Works
In HTTP-based serving, streaming is typically implemented using Server-Sent Events (SSE) or chunked transfer encoding. The serving system writes each generated token (or small group of tokens) to the HTTP response body as it becomes available, and the client reads these incremental updates in a streaming loop.
Server-Sent Events use a text-based protocol where each chunk is formatted as data: <payload>\n\n. Most LLM APIs (OpenAI, Anthropic, etc.) use SSE with JSON-encoded token payloads. The client connects to the API endpoint, reads the SSE stream line by line, parses the JSON payload from each data: line, and appends each token to the display. When the model finishes generating, the server sends a special termination event (often data: [DONE]), and the client closes the connection.
import random
import time
def simulate_streaming_decode(
prompt_tokens, n_output_tokens, tpot_ms=50.0, ttft_ms=500.0
):
"""
Simulate a streaming LLM response.
Yields (token_text, elapsed_ms) tuples as each token becomes available.
"""
# Prefill phase: simulate TTFT
time.sleep(ttft_ms / 1000.0)
elapsed = ttft_ms
vocab = [
"the",
"model",
"generates",
"tokens",
"one",
"by",
"step",
"using",
"cache",
]
for i in range(n_output_tokens):
token = random.choice(vocab)
time.sleep(tpot_ms / 1000.0)
elapsed += tpot_ms
yield token, elapsedFirst token at: 350 ms (TTFT) Final token at: 1300 ms (end-to-end latency) Average TPOT: 50.0 ms Sample tokens: model the step tokens tokens generates model efficiently ...
The simulation illustrates the two-phase structure of every streaming response. The client waits for TTFT (the prefill phase), then receives tokens at the TPOT rate. End-to-end latency is the sum of both phases, but the user's perceived experience depends heavily on TTFT since that is when the response begins to appear.
Chunking and Detokenization
A subtle issue in streaming is that tokens do not always align with visible characters. A word like "unbelievable" might tokenize as "un", "believ", "able", three separate tokens. Streaming each token independently would show "un" first, then "believ", then "able", which looks unnatural in a character-based display. Many production systems buffer a small number of tokens (or characters) before flushing to the client to create a more natural visual flow.
A more significant issue is multi-byte Unicode characters. The UTF-8 encoding of many non-ASCII characters spans multiple bytes, and a byte-level tokenizer might split a character across two tokens. Sending a partial character to the client causes a decoding error. Serving systems must either use character-level streaming with proper byte buffering, or operate at the token level and handle multi-byte characters explicitly.
Special characters in Markdown and code also require careful handling in streaming contexts. If the model is generating a code block that begins with triple backticks, the client renderer may not know how to render the content until the closing backticks arrive. Some serving systems send raw tokens while others implement client-side buffering for structured content like tables, code blocks, and LaTeX equations. The trade-off is between immediate visual feedback and correct rendering of complex content.
Streaming and Latency SLOs
Streaming changes how latency service level objectives (SLOs) are defined. Without streaming, the SLO is straightforward: 95% of requests must complete in under seconds. With streaming, you typically define two SLOs:
- P95 TTFT: 95% of requests must produce their first token within ms
- P95 TPOT: 95% of inter-token intervals must be under ms
These two SLOs can be in tension. Strategies that reduce TTFT (like prioritizing prefill work) can increase TPOT because decode steps are delayed while prefill runs. The chunked prefill technique described earlier is a direct response to this tension.
Setting appropriate SLO values requires understanding your application's user experience requirements. For a chat assistant, users notice TPOT above 80 ms and TTFT above 1 second. For an API serving developers building non-interactive pipelines, end-to-end latency matters more than TTFT or TPOT individually. For a code autocomplete tool (where the model is completing partial lines as the user types), TTFT must be extremely low (under 200 ms) and TPOT is secondary because completions are usually short. The product's user experience determines the SLO structure alongside engineering constraints.