Code Completion › Completion Latency

Latency is the invisible gating factor for code completion quality. A model that produces perfect suggestions but takes three seconds to respond is worse than a model that produces good-but-not-perfect suggestions in 200 milliseconds, because developers will not wait. Studies of IDE usage show that developers typically dismiss a completion that has not appeared within 300-500 milliseconds of triggering the request. This hard deadline shapes every engineering decision in the completion pipeline.

The 400ms threshold is not arbitrary. It corresponds roughly to the threshold at which a response begins to feel like a delay rather than an instantaneous event, as established by decades of human-computer interaction research. For keyboard-driven tasks where the developer's hands are already on the keys and their attention is on the screen, tolerances are even tighter than for mouse-driven tasks. Developers who cannot see a completion quickly enough simply keep typing, bypassing the completion system entirely. A feature that is slower than the developer's own typing speed is a feature that will not be used.

The total latency for a completion request has several components, each amenable to different optimizations:

• Retrieval latency: The time to query the cross-file index and retrieve relevant snippets. Can range from 10ms (in-memory BM25 index) to 100ms or more (remote vector store with slow embedding).
• Context assembly latency: The time to tokenize and truncate the context, including retrieved snippets. Usually small (10-30ms) but can grow with long contexts.
• Network latency: If the model runs remotely, the round-trip time to send the request and receive the first token. Typically 20-100ms for geographically distributed deployments.
• Time to first token (TTFT): The time from when the model starts processing to when it outputs the first token. Dominated by the prefill phase, which processes the full input context in parallel.
• Token generation time: The time to generate all tokens in the completion, one by one. Proportional to the number of output tokens.

For single-line completions (5-20 tokens), the prefill and network latencies dominate. For multi-line completions (50-200 tokens), token generation time becomes more significant. This asymmetry changes system design priorities: optimizing prefill helps all completions, while optimizing generation speed helps most for long completions.

Understanding where time is spent in each request type allows teams to make targeted investments. If profiling reveals that 60% of latency for short completions comes from network round-trips, the answer is to move the model closer to the user (edge deployment or on-device inference). If 60% of latency for long completions comes from token generation, the answer is speculative decoding or a faster serving infrastructure.

Speculative decoding is a technique that accelerates token generation without changing the output distribution. The idea is to use a small, fast draft model to propose $k$ tokens speculatively, then use the large target model to verify all $k$ tokens in a single forward pass.

To understand why this helps, recall that large language model inference is constrained by memory bandwidth rather than compute for the generation phase. Each generated token requires a full forward pass through the model, but that pass is heavily underutilized because it processes only one token at a time. Speculative decoding amortizes this overhead by processing a batch of $k$ tokens in a single pass, even though the tokens were generated by a draft model rather than the target.

More precisely, the draft model generates $k$ candidate tokens $\hat{c}_1, \hat{c}_2, \ldots, \hat{c}_k$ in $k$ serial steps. The target model then processes the original context plus all $k$ draft tokens in a single parallel forward pass, producing $k+1$ probability distributions. Each draft token is accepted or rejected based on whether the target model agrees with the draft model's choice, using a specific acceptance criterion that preserves the target model's output distribution:

$$\text{accept } \hat{c}_j \text{ if } u_j \leq \frac{P_{\text{target}}(\hat{c}_j)}{P_{\text{draft}}(\hat{c}_j)}$$

where:

• $\hat{c}_j$: the $j$-th token proposed by the draft model
• $u_j$: a uniform random sample from $[0, 1]$
• $P_{\text{target}}(\hat{c}_j)$: the target model's probability for this token
• $P_{\text{draft}}(\hat{c}_j)$: the draft model's probability for this token

The acceptance criterion is a ratio test: if the target model considers a token at least as probable as the draft model, accept it deterministically. If the target model is less confident than the draft, accept it with probability proportional to the ratio. This ensures that the final output distribution is exactly the target model's distribution, regardless of the draft model's quality. A poor draft model simply causes more rejections and reduces the speedup, but never degrades output quality below the target model's level.

When the draft model is wrong, the target model's distribution is used to produce the correct token and the speculative chain stops. On average, if the draft model agrees with the target model for most tokens (which is common when the draft model is a smaller version of the same model family), the effective throughput is much higher than running the target model alone.

For code completion, speculative decoding can reduce generation time by 2-4x on typical short completions because code is often locally predictable: given a few tokens of context, the next tokens are highly constrained by syntax and common patterns that even small models learn well. A draft model that has seen enough code to predict common variable names, common method calls, and common control flow patterns will agree with the large model on a majority of tokens, enabling the large model to verify and confirm the draft in far fewer total forward passes.

Large language models maintain a key-value (KV) cache that stores the intermediate attention keys and values for all tokens in the context. Recomputing these from scratch for every completion request is expensive. The KV cache allows the model to skip recomputation for tokens it has already processed.

In a completion system, this matters most for the stable parts of the context. If the developer is editing a file and triggering completions every few seconds, the top of the file (imports, class definitions, earlier functions) changes rarely. A prefix caching system that stores the KV cache for the stable prefix and reuses it across requests can dramatically reduce effective prefill cost.

The mechanics of prefix caching require matching the beginning of a new request's token sequence against previously cached sequences. If the first 1000 tokens of a new request are identical to a recently cached request (which happens frequently when the developer is editing code near the end of a long file), the prefill computation for those 1000 tokens can be entirely skipped. Only the new tokens (those that changed since the last request) need to be processed.

Systems like RadixAttention (used in vLLM's prefix caching) organize the KV cache as a trie indexed by token sequences, allowing efficient lookup and reuse of shared prefixes across concurrent requests from multiple developers using the same codebase. In a large deployment, many developers working on the same repository will have overlapping prefixes (the same shared library files, the same common utility modules), making cross-request cache sharing a significant multiplier on effective throughput.

Prefix caching is especially powerful in combination with the context assembly strategies discussed earlier. If the system consistently prepends retrieved snippets in the same order, and if those snippets change slowly (they come from stable library files), the retrieved context portion of the prefix can be cached even as the per-developer suffix changes. Designing the context assembly pipeline to maximize prefix stability is a subtle but high-impact optimization.

Even with latency optimizations, there is a gap between when the first token is ready and when the full completion is available. Streaming delivery shows the completion token by token as the model generates it, rather than waiting for the full sequence.

For multi-line completions, streaming is particularly valuable because the developer can start reading and evaluating the suggestion while the model is still generating the tail. If the first few lines look wrong, they can dismiss the suggestion early without waiting for full generation. This behavior means that the perceived latency for multi-line completions is dominated by the time to first token rather than the total generation time, partially compensating for the longer generation times of large completions.

The editor integration must handle the visual presentation of partial completions gracefully, typically by rendering a "ghost text" overlay that updates incrementally and disappears when the developer types or presses Escape. The challenge is rendering partial completions that are syntactically incomplete: a streaming function body that has opened a for loop but not yet produced the closing brace is syntactically invalid, but the editor should still display it as a hint rather than hiding it until it is syntactically valid. Most production systems handle this by treating the streaming text as raw characters rather than parsed code until the full completion is available.

For short completions, total latency is well within the developer tolerance threshold. For function-body completions, latency exceeds 600ms, making streaming delivery essential so developers see early tokens while the model finishes.

Every keystroke can in principle trigger a new completion request, but most keystrokes are not meaningful trigger points. When a developer is typing quickly, the context changes faster than requests can be fulfilled, making in-flight requests immediately stale. Debouncing suppresses completion requests until the developer pauses for a minimum duration (typically 50-150ms), coalescing rapid sequences of keystrokes into a single request.

Debouncing dramatically reduces the request rate and prevents the model server from being overwhelmed with stale requests. The tradeoff is that it adds a minimum latency to every completion: a developer who pauses briefly will wait at least the debounce period before a completion appears. Systems tune the debounce threshold empirically by finding the value that minimizes completion miss rate (completions triggered too late to be useful) while keeping server load manageable.

Beyond simple debouncing, some systems implement smarter coalescing strategies. If a completion request is in-flight and the developer types two more characters, the system can cancel the in-flight request and send a new one with the updated context, rather than waiting for the stale result and then suppressing it. This speculative cancellation reduces waste but requires the server to support request cancellation, which is non-trivial with streaming responses.

The debounce threshold interacts with the completion latency in a non-obvious way. If the model typically takes 180ms to respond and the debounce threshold is 100ms, the developer will see the completion 280ms after they pause, which is within the 400ms tolerance. If the debounce threshold is raised to 200ms to reduce server load, the total wait becomes 380ms, dangerously close to the threshold. Changes to either the debounce threshold or the model's latency must be evaluated together.