Code Completion: Context, Ranking, Latency & UX

Every time a developer types a dot after a variable name and pauses, their editor faces a question that looks deceptively simple: what should come next? Code completion has been a feature of development environments for decades, first as static keyword lists, then as type-aware suggestions driven by language servers, and now as a full-blown language model inference problem. The modern incarnation, powered by large language models, can suggest entire functions, fill in argument lists, and predict multi-line logic blocks with striking accuracy. Yet delivering that capability inside a text editor, where every keystroke can trigger a new request, requires solving several hard problems simultaneously: what context should the model see, how do you rank competing suggestions, how do you keep latency low enough that the feature is not annoying, and how do you present results without disrupting the developer's flow.

The history of code completion reflects a gradual expansion of ambition. Early systems like IntelliSense in Visual Studio and content-assist in Eclipse relied entirely on static analysis: parse the file, infer the type of the expression under the cursor, look up its members, and present a sorted list. These systems were fast and exact, but they could not suggest novel code the developer had not yet written. They completed what was already known, not what the developer intended.

Neural approaches changed the game. Models trained on billions of lines of code learned statistical associations between code patterns, enabling them to suggest completions that no static analysis could produce: a loop body that fits the surrounding variable names, a function call with arguments derived from the function's documentation, or a test case that mirrors the style of existing tests in the file. With the arrival of transformer-based models like Codex, GitHub Copilot, and later DeepSeek Coder and CodeLlama, code completion shifted from a symbol-lookup problem to a generative inference problem.

This chapter covers all four of the central challenges in depth. We will look at how completion context is constructed from the raw text of an open file, how ranking separates good suggestions from plausible-but-wrong ones, what engineering decisions determine end-to-end latency, and how UX choices affect whether developers accept or ignore suggestions.

The quality of any completion model is bounded by the quality of the context it receives. Feed the model too little context and it cannot produce relevant suggestions. Feed it too much and you either exceed the model's context window, add unnecessary latency, or dilute the signal with irrelevant text. Constructing the right context is therefore one of the most consequential decisions in the entire system.

Code completion context is not simply "everything before the cursor." It is a carefully engineered view of the codebase, assembled from multiple sources and filtered for relevance. The assembler must balance recency (the most recently written code is the strongest signal), scope (the surrounding function, class, and module matter more than distant code), and diversity (sometimes an example from a completely different file is the key piece of context the model needs).

Early code completion systems only considered the text before the cursor, called the prefix. The model was asked to continue from where the developer stopped. This works when the developer is writing new code at the end of a function, but it fails badly when they are inserting code in the middle of an existing block or filling in an argument to a function call.

Consider a developer who has already written the outer structure of a function and now wants to fill in its body. With a prefix-only model, the model sees the beginning of the function but not the return statement that constrains what the body must produce. This leads to completions that are locally plausible but globally wrong: the body might manipulate the wrong variables or fail to produce a value matching the type the caller expects.

Modern systems solve this with a training objective called fill-in-the-middle (FIM). Instead of training the model only to predict continuations, FIM training masks a random span from the middle of a document and asks the model to reconstruct the missing portion given both the prefix (text before the span) and the suffix (text after the span). At inference time, the editor sends the prefix and suffix separately, and the model produces the infill that fits between them.

The FIM format rearranges the input into a structure like:

where <PRE>, <SUF>, and <MID> are special tokens the model learns to associate with their respective roles. The model then generates tokens that would naturally connect the prefix to the suffix. This matters enormously in practice because a large fraction of real completions are insertions: adding a parameter to an existing function call, writing the body of a function whose signature already exists, or filling in a missing branch of a conditional.

FIM training also changes what the model learns about code structure. With prefix-only training, the model develops a strong prior toward producing code that looks syntactically complete: it always writes closing braces, return statements, and function endings. With FIM training, the model learns to produce partial fragments that are syntactically open-ended, trusting that the suffix provides the closing structure. This seemingly small change substantially improves completion quality for insertion tasks, which in practice account for a large share of developer edits.

Research on models like InCoder and CodeLlama showed that FIM-trained models achieve significantly higher exact-match rates on insertion benchmarks than comparable prefix-only models, with the gap widening for longer insertions. The reason is intuitive: for a short one-token insertion the suffix provides little additional information, but for a multi-line insertion the suffix is an extremely strong constraint that eliminates entire classes of plausible-but-wrong completions.

The simplest completion context is the contents of the file currently open in the editor. For short files, the entire file fits in the model's context window and provides useful signals: the imports at the top tell the model what libraries are in use, the function signatures already defined tell it what API the developer is building, and the variable names in scope tell it what identifiers are available.

Short files are the easy case. The challenge arises for long files, where the developer has accumulated hundreds or thousands of lines across multiple classes and functions. Such files routinely exceed the context windows of smaller completion models (typically 2K to 8K tokens for fast inference), and even modern models with 128K context windows face a practical tradeoff: longer contexts mean higher prefill latency.

For long files that exceed the context window, the system must decide which portions to include. A common strategy is to always include the full prefix up to the cursor (since the most recent code is the strongest signal), then fill remaining context tokens with the suffix starting from the cursor, then optionally include earlier sections of the file if budget remains. This truncation heuristic encodes a model of relevance. The code immediately around the cursor is more relevant than code far away, the current function is more relevant than unrelated functions, and recently modified code is more relevant than stable legacy code.

An alternative approach, used in some production systems, is to build a per-file index that identifies the most relevant fragments to include even if they are far from the cursor. For example, if the cursor is inside a method that calls several helper functions defined earlier in the same file, including those helper function signatures (even if their bodies are omitted) provides the model with the type and interface information it needs without consuming the entire context budget.

A single file is rarely sufficient. Modern codebases are modular: a developer writing a new function might need context from type definitions in one file, utility functions in another, and interface contracts in a third. Without cross-file context, the model cannot see the types of the arguments it is supposed to use, the methods available on the objects it is working with, or the conventions established by adjacent code.

Cross-file context requires a retrieval step. The system builds an index of the workspace (often a vector store or a BM25-style inverted index of identifiers and snippets) and queries it at completion time with the local context as a probe. The retrieved snippets are then prepended to the completion request as additional context.

Several retrieval signals are useful:

• Import graph proximity: Files imported by the current file are likely more relevant than files elsewhere in the project. If the current file imports auth.utils, then the contents of auth/utils.py are immediately relevant.
• Identifier overlap: Files that define identifiers referenced in the current file are highly relevant. If the cursor is inside a call to UserRepository, the file that defines UserRepository should be in context.
• Recency: Files the developer has recently edited or viewed are often related to the current task. Most developers work on a small set of files at a time, and recently opened files are likely thematically related.
• Semantic similarity: Embedding-based similarity can surface relevant code that shares no common identifiers but solves a similar problem. This is particularly valuable for surfacing patterns from elsewhere in the codebase that the developer might want to follow.

The challenge is that retrieving and embedding snippets takes time. Production systems use aggressive caching: index updates happen in the background as files change, and common queries are cached for a short window so that nearby keystrokes reuse the same retrieved context rather than triggering fresh retrieval. The index itself is often a combination of a fast lexical index for identifier lookup and a slower but more powerful vector index for semantic similarity.

The architecture of cross-file retrieval resembles the retrieval-augmented generation (RAG) pipelines we examined earlier in this book, adapted for code. The key differences are that the documents (code files) have strong structural constraints (syntax, scope, module boundaries), the queries (the local context around the cursor) are code rather than natural language, and latency requirements are far stricter than in typical question-answering RAG systems.

Beyond individual files, some completion systems attempt to understand the broader structure of a repository: its conventions for naming, its test patterns, its error handling idioms, and its architectural boundaries. This is sometimes called repository-level context or long-context completion.

One approach encodes a summary of the repository into a persistent context that is prepended to every completion request. The summary might include the repository's main modules, common patterns extracted from the codebase, and project-level configuration (such as which version of a library is being used, which could affect API availability). Another approach uses a retrieval-augmented generation (RAG) setup with chunks of the repository indexed by a code-aware embedding model.

Repository-level context is particularly valuable for large, mature codebases where the developer is expected to follow established patterns. A model that knows the repository uses a specific logging framework, a specific error-handling convention, and a specific style for writing tests will produce far more useful suggestions than one that relies only on general training data. This is one reason that enterprise code completion systems often outperform general-purpose completions for developers working in large internal codebases.

The practical limit here is latency. Encoding long contexts takes time, and retrieval introduces its own delay. Systems that make aggressive use of repository context typically do so in a background prefetch mode, building up a cached representation that is ready when the developer triggers a completion. The cached representation is invalidated when files change, triggering a background refresh that completes before the developer's next completion request.

Given a fixed context window size (say, 8192 tokens for a smaller model or 128K for a larger one), how should the budget be allocated across prefix, suffix, and retrieved snippets?

The allocation problem optimizes the expected completion quality given the model's training distribution and the typical distribution of completion requests. A model that was trained primarily on single-file contexts will benefit less from cross-file snippets than one specifically trained with repository context. A model with a large context window can afford to be generous with retrieved snippets, while a small model must be much more selective.

A typical allocation strategy looks like this:

• Prefix: Allocate the largest share, often 50-70% of the budget. The most recent code is the strongest signal.
• Suffix: Allocate 15-25% of the budget. Useful for FIM but less critical than prefix.
• Retrieved snippets: Allocate the remainder. Sort retrieved snippets by relevance score and include as many as fit.

This is not a universal formula. The right allocation depends on the model's training distribution, the average file length in the target language, and empirical evaluation on completion benchmarks. Systems that have been deployed at scale typically discover through A/B testing that their initial allocation assumptions were suboptimal, and iterate toward data-driven allocation policies.

As models grow larger and support longer context windows, it becomes practical to include more cross-file retrieval. Smaller models must be more selective, prioritizing the immediate prefix because they have demonstrated through training that local context is more reliably useful than distant snippets.

Given a completion context, the model produces a probability distribution over possible next tokens at each step. When using a model that generates completions autoregressively, the full sequence is decoded and returned as a suggestion. But often, the system generates multiple candidate completions and must decide which one to show the developer first.

Ranking is the problem of ordering candidate completions by their expected quality. A completion that is syntactically correct, semantically appropriate, and matches the developer's intent should rank above one that is merely probable under the model. This distinction matters because the model's probability distribution reflects patterns in training data, not the developer's actual intent in the current moment. A highly probable completion might be a common boilerplate that is wrong for this specific use case, while a slightly less probable completion might be exactly what the developer needs.

The simplest ranking signal is the model's own probability score for the generated sequence. For a completion $c = (c_1, c_2, \ldots, c_T)$, the log-probability is:

where:

• $c_t$: the $t$-th token in the completion
• $T$: the length of the completion in tokens
• $\text{context}$: the prefix, suffix, and any retrieved snippets
• $\log P(c_t \mid \cdots)$: the model's log-probability for token $c_t$ given all preceding context

The raw log-probability sums across all tokens, which means longer completions are systematically penalized: a completion of 20 tokens will almost always have a lower total log-probability than a 5-token completion, even if every token in the 20-token completion is equally probable. This makes length-normalized comparison essential.

Longer completions tend to have lower log-probabilities simply because they contain more tokens, each of which contributes a multiplicative penalty. To compare completions of different lengths fairly, systems typically use the mean log-probability:

This length-normalized score penalizes completions with many low-probability tokens relative to completions where every token is highly confident. A completion where every token has mean log-probability of -0.2 is more confident than one where some tokens have log-probability -2.0, even if the overall sequence length is the same.

Log-probability has important limitations as a ranking signal. The model's probability distribution is calibrated to its training data distribution, not to the developer's immediate intent. A completion that is common in training data (and therefore high-probability) might be exactly what we do not want here. For example, if the model has seen many Python files that use assert x > 0 for input validation, it might assign high probability to that pattern even when the developer is writing production code where exceptions are more appropriate. Log-probability captures "how typical is this" rather than "how correct is this for this particular situation."

Before ranking, candidates are filtered for basic validity. A completion that fails to parse as valid syntax in the target language is almost certainly wrong, even if the model assigned it a high probability. Common filters include:

• Syntax validity: Parse the completion and reject completions with syntax errors. This is fast in most languages using incremental parsers. Tree-sitter, a popular incremental parsing library used in many editors, can parse partial code fragments and identify syntax errors in under a millisecond.
• Type consistency: In statically typed languages, check that the completion does not introduce obvious type errors. This is harder and typically requires integration with the language server. Systems that have access to a running language server can query it for type information at low latency.
• Identifier existence: Check that identifiers referenced in the completion are in scope. Hallucinated method names or nonexistent variables are a common failure mode, particularly for APIs the model has not seen frequently in training.
• Style matching: Strip completions that violate project-level formatting rules (trailing whitespace, incorrect indentation style). While these are cosmetic, accepting a completion that immediately triggers a linting error creates a negative experience.

These filters reduce the candidate set before expensive reranking, improving both quality and latency. The key insight is that syntactic filters are fast (sub-millisecond) and catch a disproportionate fraction of bad completions, while type-based filters are slower but catch more subtle errors.

The primary completion model is often large, making it expensive to generate many diverse candidates. A common pipeline uses the large model to generate a small number of candidates (say, three to five), then uses a separate smaller reranker model to score them and reorder the list.

The reranker is typically a sequence classifier or a smaller language model fine-tuned to predict acceptance probability. It receives the context and a candidate completion as input and outputs a scalar score. Because the reranker only scores rather than generates, it is much faster than the generation model and can be run in parallel on all candidates.

Fine-tuning the reranker on real acceptance data (completions accepted by developers versus dismissed ones) grounds the scoring in observable human behavior rather than proxy metrics like log-probability. A developer who types the exact same code that a dismissed completion suggested is providing a clear signal: the suggestion was correct, but perhaps it appeared at the wrong moment or was visually presented in a way that caused the developer to miss it. A developer who accepts a completion and immediately backspaces is providing the opposite signal. These timing and correction signals can be learned by a reranker trained on large logs of developer interactions.

One important consideration in reranker design is the distribution shift between training and deployment. Rerankers trained on historical acceptance data inherit any biases in what completions were shown historically. If the system previously favored certain completion styles, the reranker will have more training signal for those styles and may unfairly downrank novel completion types that were rarely shown. Mitigating this requires deliberate data diversity in training or explicit correction for historical presentation bias.

When a developer accepts a completion, that is a strong positive signal. When they dismiss it or immediately type over it, that is a signal it was not useful. Collecting these acceptance and rejection events and using them to fine-tune both the generation model and the reranker closes the feedback loop between the deployed system and real developer behavior.

This is an instance of reinforcement learning from human feedback (RLHF) applied at the system level rather than the conversation level. The acceptance event plays the role of the human preference signal, and the completion model is updated to produce more accepted suggestions over time. The key difference from conversational RLHF is that the feedback is implicit (the developer does not rate the suggestion) and abundant (millions of completion events per day in large deployments).

The practical challenge is selection bias: developers only see the top-ranked completion, so you observe acceptance or rejection of that completion but not of alternatives. If you knew the developer would have accepted completion B but you showed them completion A, you never learn that signal. Systems that occasionally surface lower-ranked completions (with deliberate exploration) can gather richer training data, though this must be balanced against user experience. Exploring too frequently degrades the product quality that real users experience; exploring too rarely leads to a model that is overfit to the historical top-ranked distribution.

Some systems address this by running A/B experiments at the user level rather than the suggestion level: a randomly selected subset of users receives an experimental ranking model, and their aggregate acceptance rates provide a signal about which ranking strategy performs better without degrading every individual's experience.

The reranker often surfaces different top candidates than raw log-probability, showing that developer-accepted completions do not always coincide with the model's most probable output. The winning candidate under the reranker may be the one that best fits the developer's observed behavioral patterns, even if the model's generative probability slightly prefers an alternative.

Latency is the invisible gating factor for code completion quality. A model that produces perfect suggestions but takes three seconds to respond is worse than a model that produces good-but-not-perfect suggestions in 200 milliseconds, because developers will not wait. Studies of IDE usage show that developers typically dismiss a completion that has not appeared within 300-500 milliseconds of triggering the request. This hard deadline shapes every engineering decision in the completion pipeline.

The 400ms threshold is not arbitrary. It corresponds roughly to the threshold at which a response begins to feel like a delay rather than an instantaneous event, as established by decades of human-computer interaction research. For keyboard-driven tasks where the developer's hands are already on the keys and their attention is on the screen, tolerances are even tighter than for mouse-driven tasks. Developers who cannot see a completion quickly enough simply keep typing, bypassing the completion system entirely. A feature that is slower than the developer's own typing speed is a feature that will not be used.

The total latency for a completion request has several components, each amenable to different optimizations:

• Retrieval latency: The time to query the cross-file index and retrieve relevant snippets. Can range from 10ms (in-memory BM25 index) to 100ms or more (remote vector store with slow embedding).
• Context assembly latency: The time to tokenize and truncate the context, including retrieved snippets. Usually small (10-30ms) but can grow with long contexts.
• Network latency: If the model runs remotely, the round-trip time to send the request and receive the first token. Typically 20-100ms for geographically distributed deployments.
• Time to first token (TTFT): The time from when the model starts processing to when it outputs the first token. Dominated by the prefill phase, which processes the full input context in parallel.
• Token generation time: The time to generate all tokens in the completion, one by one. Proportional to the number of output tokens.

For single-line completions (5-20 tokens), the prefill and network latencies dominate. For multi-line completions (50-200 tokens), token generation time becomes more significant. This asymmetry changes system design priorities: optimizing prefill helps all completions, while optimizing generation speed helps most for long completions.

Understanding where time is spent in each request type allows teams to make targeted investments. If profiling reveals that 60% of latency for short completions comes from network round-trips, the answer is to move the model closer to the user (edge deployment or on-device inference). If 60% of latency for long completions comes from token generation, the answer is speculative decoding or a faster serving infrastructure.

Speculative decoding is a technique that accelerates token generation without changing the output distribution. The idea is to use a small, fast draft model to propose $k$ tokens speculatively, then use the large target model to verify all $k$ tokens in a single forward pass.

To understand why this helps, recall that large language model inference is constrained by memory bandwidth rather than compute for the generation phase. Each generated token requires a full forward pass through the model, but that pass is heavily underutilized because it processes only one token at a time. Speculative decoding amortizes this overhead by processing a batch of $k$ tokens in a single pass, even though the tokens were generated by a draft model rather than the target.

More precisely, the draft model generates $k$ candidate tokens $\hat{c}_1, \hat{c}_2, \ldots, \hat{c}_k$ in $k$ serial steps. The target model then processes the original context plus all $k$ draft tokens in a single parallel forward pass, producing $k+1$ probability distributions. Each draft token is accepted or rejected based on whether the target model agrees with the draft model's choice, using a specific acceptance criterion that preserves the target model's output distribution:

where:

• $\hat{c}_j$: the $j$-th token proposed by the draft model
• $u_j$: a uniform random sample from $[0, 1]$
• $P_{\text{target}}(\hat{c}_j)$: the target model's probability for this token
• $P_{\text{draft}}(\hat{c}_j)$: the draft model's probability for this token

The acceptance criterion is a ratio test: if the target model considers a token at least as probable as the draft model, accept it deterministically. If the target model is less confident than the draft, accept it with probability proportional to the ratio. This ensures that the final output distribution is exactly the target model's distribution, regardless of the draft model's quality. A poor draft model simply causes more rejections and reduces the speedup, but never degrades output quality below the target model's level.

When the draft model is wrong, the target model's distribution is used to produce the correct token and the speculative chain stops. On average, if the draft model agrees with the target model for most tokens (which is common when the draft model is a smaller version of the same model family), the effective throughput is much higher than running the target model alone.

For code completion, speculative decoding can reduce generation time by 2-4x on typical short completions because code is often locally predictable: given a few tokens of context, the next tokens are highly constrained by syntax and common patterns that even small models learn well. A draft model that has seen enough code to predict common variable names, common method calls, and common control flow patterns will agree with the large model on a majority of tokens, enabling the large model to verify and confirm the draft in far fewer total forward passes.

Large language models maintain a key-value (KV) cache that stores the intermediate attention keys and values for all tokens in the context. Recomputing these from scratch for every completion request is expensive. The KV cache allows the model to skip recomputation for tokens it has already processed.

In a completion system, this matters most for the stable parts of the context. If the developer is editing a file and triggering completions every few seconds, the top of the file (imports, class definitions, earlier functions) changes rarely. A prefix caching system that stores the KV cache for the stable prefix and reuses it across requests can dramatically reduce effective prefill cost.

The mechanics of prefix caching require matching the beginning of a new request's token sequence against previously cached sequences. If the first 1000 tokens of a new request are identical to a recently cached request (which happens frequently when the developer is editing code near the end of a long file), the prefill computation for those 1000 tokens can be entirely skipped. Only the new tokens (those that changed since the last request) need to be processed.

Systems like RadixAttention (used in vLLM's prefix caching) organize the KV cache as a trie indexed by token sequences, allowing efficient lookup and reuse of shared prefixes across concurrent requests from multiple developers using the same codebase. In a large deployment, many developers working on the same repository will have overlapping prefixes (the same shared library files, the same common utility modules), making cross-request cache sharing a significant multiplier on effective throughput.

Prefix caching is especially powerful in combination with the context assembly strategies discussed earlier. If the system consistently prepends retrieved snippets in the same order, and if those snippets change slowly (they come from stable library files), the retrieved context portion of the prefix can be cached even as the per-developer suffix changes. Designing the context assembly pipeline to maximize prefix stability is a subtle but high-impact optimization.

Even with latency optimizations, there is a gap between when the first token is ready and when the full completion is available. Streaming delivery shows the completion token by token as the model generates it, rather than waiting for the full sequence.

For multi-line completions, streaming is particularly valuable because the developer can start reading and evaluating the suggestion while the model is still generating the tail. If the first few lines look wrong, they can dismiss the suggestion early without waiting for full generation. This behavior means that the perceived latency for multi-line completions is dominated by the time to first token rather than the total generation time, partially compensating for the longer generation times of large completions.

The editor integration must handle the visual presentation of partial completions gracefully, typically by rendering a "ghost text" overlay that updates incrementally and disappears when the developer types or presses Escape. The challenge is rendering partial completions that are syntactically incomplete: a streaming function body that has opened a for loop but not yet produced the closing brace is syntactically invalid, but the editor should still display it as a hint rather than hiding it until it is syntactically valid. Most production systems handle this by treating the streaming text as raw characters rather than parsed code until the full completion is available.

For short completions, total latency is well within the developer tolerance threshold. For function-body completions, latency exceeds 600ms, making streaming delivery essential so developers see early tokens while the model finishes.

Every keystroke can in principle trigger a new completion request, but most keystrokes are not meaningful trigger points. When a developer is typing quickly, the context changes faster than requests can be fulfilled, making in-flight requests immediately stale. Debouncing suppresses completion requests until the developer pauses for a minimum duration (typically 50-150ms), coalescing rapid sequences of keystrokes into a single request.

Debouncing dramatically reduces the request rate and prevents the model server from being overwhelmed with stale requests. The tradeoff is that it adds a minimum latency to every completion: a developer who pauses briefly will wait at least the debounce period before a completion appears. Systems tune the debounce threshold empirically by finding the value that minimizes completion miss rate (completions triggered too late to be useful) while keeping server load manageable.

Beyond simple debouncing, some systems implement smarter coalescing strategies. If a completion request is in-flight and the developer types two more characters, the system can cancel the in-flight request and send a new one with the updated context, rather than waiting for the stale result and then suppressing it. This speculative cancellation reduces waste but requires the server to support request cancellation, which is non-trivial with streaming responses.

The debounce threshold interacts with the completion latency in a non-obvious way. If the model typically takes 180ms to respond and the debounce threshold is 100ms, the developer will see the completion 280ms after they pause, which is within the 400ms tolerance. If the debounce threshold is raised to 200ms to reduce server load, the total wait becomes 380ms, dangerously close to the threshold. Changes to either the debounce threshold or the model's latency must be evaluated together.

The best completion model in the world can fail to help developers if the suggestions are presented poorly. UX design for code completion is a careful balance between surfacing relevant suggestions and not distracting the developer from their current thought. Every design decision in this space reflects a fundamental tension: the more aggressively you surface suggestions, the more useful the system is when the suggestion is right, but the more disruptive it is when the suggestion is wrong.

The dominant UX pattern for modern LLM-based completion is ghost text: the suggested completion appears in-place at the cursor in a visually distinct style (typically grayed out or slightly transparent) as if it were already typed but not yet committed. The developer can accept the full suggestion by pressing Tab, reject it by continuing to type, or partially accept it (in systems that support it) by pressing a modifier key.

Ghost text is powerful because it is non-disruptive. The developer does not have to navigate to a dropdown or move their hands to a different control. If the suggestion is relevant, accepting it is a single keystroke. If it is not, it costs no extra effort beyond the keystrokes the developer was going to type anyway. This cost asymmetry matters: the downside of a bad suggestion is near zero, while the upside of a good one is significant.

The visual design of ghost text requires care. If it is too subtle, developers miss relevant suggestions. If it is too prominent, it becomes visually distracting when the suggestion is wrong. Most systems use a light gray color for ghost text, relying on the contrast with the editor's normal text color to make suggestions readable but non-intrusive. Font styling (italic, lighter weight) can reinforce the "provisional" nature of the suggestion without requiring strong color contrast.

Ghost text also creates a subtle psychological effect: by displaying the suggestion in-place, it makes accepting the suggestion feel like confirming rather than inserting. Developers often report that accepting a correct ghost text completion feels effortless in a way that selecting from a dropdown does not, even though the keystrokes required are similar. This perceived effortlessness increases adoption.

Single-line completions (completing the remainder of the current line) have a lower perceived risk for developers. The suggestion is small, easy to evaluate at a glance, and quick to dismiss if wrong. Multi-line completions (suggesting entire function bodies or code blocks) offer higher potential value but introduce new UX challenges.

A multi-line completion that is partially correct but partially wrong creates a new problem: the developer must edit the accepted suggestion rather than write from scratch. Whether this is faster than typing from scratch depends on how much editing is required. Systems that offer partial acceptance (accepting up to the end of the first line, or up to the first syntax unit) give developers finer control.

The cognitive cost of evaluating a multi-line suggestion is higher than for a single-line suggestion. A developer must read several lines, understand the proposed logic, identify any errors, and decide whether to accept or reject. For a 10-line function body, this evaluation might take 5-15 seconds, during which the developer is reading rather than writing. For suggestions that are mostly right (needing only minor tweaks), this is a worthwhile investment. For suggestions that are mostly wrong, it is pure overhead.

Automatic ghost-text multi-line completions are most useful when:

• The developer has written a function signature and the body is predictable from the signature and surrounding context.
• The developer is implementing a well-known pattern (sorting, filtering, error handling) that the model has seen in many forms.
• The completion is short enough to read in under two seconds.

They are less useful for novel algorithms, business logic, or any code where the developer has not yet decided on the implementation approach. In those cases, the completion interrupts the developer's thinking rather than supporting it.

Some systems adapt their behavior based on inferred task type. Boilerplate detection heuristics (looking for patterns like "getter/setter", "constructor", "toString", or similar language-specific idioms) can identify high-confidence contexts where multi-line completions are likely to be correct, and route those requests to a larger, slower model while using a faster model for less predictable completions.

Not every typing event should trigger a completion attempt. Showing suggestions on every keystroke would be visually overwhelming and would surface many irrelevant suggestions. Production systems use trigger conditions to decide when to initiate a request:

• After whitespace: Pressing space or Enter after a keyword is a strong signal that the developer is starting a new construct.
• After an opening delimiter: An open parenthesis suggests an argument list is being started, an open brace suggests a block body.
• After a dot: Accessing a member of an object is a classic completion trigger, inherited from the original IDE completion paradigm.
• After a pause: If the developer stops typing for more than the debounce threshold, that suggests they are thinking and may benefit from a suggestion.
• Explicit invocation: Many systems allow manual trigger with a keyboard shortcut (Ctrl+Space or similar) for on-demand completions.

Systems also learn negative trigger conditions: if a developer has recently dismissed a suggestion at a particular cursor position, do not immediately re-trigger at the same position. This avoids the annoying behavior where the same unwanted suggestion keeps reappearing as the developer tries to dismiss it and type something different.

Trigger condition tuning is a product discipline as much as a technical one. The right set of triggers depends on the programming language (Python's dynamic typing makes member-access triggers less reliable than in Java), the editor's existing completion features (a language server may already handle symbol completion, leaving LLM completions to focus on multi-token suggestions), and the developer population's preferences (experienced developers often prefer explicit invocation over automatic triggers).

The acceptance rate for completion suggestions varies widely across tool types and code styles. Research from GitHub Copilot and similar systems suggests acceptance rates typically range from 20% to 35% for shown completions, with significant variation by:

• Language: Boilerplate-heavy languages (Java, C#) see higher acceptance rates than expressive languages (Python, Ruby) because boilerplate is more predictable.
• Task type: Implementing well-known patterns sees higher acceptance. Novel logic sees lower acceptance.
• Developer experience: More experienced developers are more selective, accepting completions that are closer to what they would have written.
• Completion length: Shorter completions have higher acceptance rates per suggestion.

Java and C# show higher acceptance rates because their verbose patterns (getter/setter boilerplate, interface implementations, XML documentation comments) are highly predictable and closely match the model's training data. Python and Ruby show lower rates because the language's expressiveness means there are more valid ways to write the same logic, making any particular suggestion less likely to match the developer's intent exactly.

Beyond accept/dismiss, production systems benefit from richer feedback signals:

• Edit after accept: If a developer accepts a completion and immediately edits it, the original suggestion needed improvement. Systems can track what was edited to identify systematic model weaknesses.
• Re-type after dismiss: If a developer dismisses a suggestion and then types exactly what the model suggested, the suggestion was correct but perhaps poorly timed or presented.
• Copy from suggestion: Some systems detect when a developer copies text that appeared in a dismissed suggestion, indicating partial utility.

These signals feed back into model fine-tuning pipelines, creating a continuous improvement loop between the deployed system and real developer behavior. The richest signal, however, comes from downstream outcomes: did the code that was accepted (or manually typed) work correctly? Did it require debugging? Did it appear in a subsequent bug report? These downstream signals are hard to attribute to specific completion events, but aggregating them over large populations can reveal which types of completions correlate with code quality outcomes.

To make these concepts concrete, let's build a minimal code completion system. We will implement context assembly, simple candidate scoring, and a simulation of the full completion pipeline.

We start by importing the libraries we need.

The context assembler takes the full file text and cursor position, and returns a trimmed context that fits within a token budget.

The assembler correctly keeps the recent prefix, includes the suffix starting from the cursor, and fits the retrieved snippet within the remaining budget. Notice that the prefix is truncated from the left (discarding distant earlier code) rather than from the right, so the code immediately before the cursor is always preserved.

Next, we implement the length-normalized log-probability scoring function and a simple filter for identifier validity.

The scoring correctly places high-confidence completions at the top, while the hallucinated compute_std function is filtered out because the identifier is not in scope. The filter catches the most egregious hallucination (a completely nonexistent function name) before any more expensive reranking steps are applied.

Finally, let's simulate the end-to-end latency model to understand how the different components stack up.

Combining prefix caching and speculative decoding keeps latency under 400ms for completions up to about 50 tokens. For longer completions (function bodies), streaming delivery is essential to avoid blocking the developer while still surfacing useful suggestions.

Understanding where current completion systems came from helps explain why they are designed the way they are, and what constraints shaped their architecture.

The first generation of IDE completion in the 1990s and early 2000s was purely symbol-based. IntelliSense, introduced in Visual Studio in 1996, worked by parsing the code, building a symbol table, and presenting a dropdown list of valid symbols at each completion point. This approach was perfectly accurate for the symbols it knew about, but it could not suggest anything the developer had not already written. It was a lookup system, not a prediction system.

The second generation, emerging in the late 2000s and early 2010s, added probabilistic models trained on code corpora. Tools like Eclipse's content assist began incorporating frequency-based ranking: symbols that appeared more often in the codebase or in training data were ranked higher. This improved the ordering of the static symbol list but still did not enable generative completions.

The third generation arrived with neural models around 2016-2018. Tools like Kite and TabNine began using neural sequence models trained on large code corpora to predict the next token or tokens. These models could suggest novel code, not just look up existing symbols. However, the models were small by today's standards, and their suggestions were often syntactically correct but semantically shallow.

The fourth and current generation, beginning with GitHub Copilot's launch in 2021, uses large transformer models (initially Codex, a GPT-3 descendant fine-tuned on code) capable of understanding broad context and generating coherent multi-line completions. The jump in quality from third to fourth generation was dramatic, showing the general scaling benefits seen in other language model applications. Copilot's public launch marked the moment when LLM-based code completion became a mainstream developer tool rather than a research curiosity.

Each generation introduced new challenges. Symbol-based completion required maintaining accurate symbol tables. Statistical models required managing training data pipelines and model updates. Neural completions required balancing model size against latency. LLM-based completions required solving the full stack problem: retrieval, context assembly, ranking, latency optimization, and UX design simultaneously.

Code completion with large language models works remarkably well for common patterns but has systematic failure modes that are important to understand.

Hallucination of identifiers and APIs. Language models confidently suggest method calls, class names, and function signatures that do not exist. This happens because the model has learned statistical associations between API patterns in its training data, not an understanding of the actual API surface. A model trained on Python code before a library's major version change may suggest deprecated or renamed methods. The requests library's migration from requests.get() to various async patterns, or changes in deep learning framework APIs between major versions, are examples of exactly this failure. Post-processing filters that check identifier existence help but cannot catch all cases, especially for user-defined APIs the model has not seen.

Context length limits and the lost-in-the-middle problem. Even with large context windows, models attend most strongly to the beginning and end of the context. Code buried in the middle of a long context window is often effectively ignored. Research on the "lost-in-the-middle" phenomenon showed that retrieval-augmented models perform significantly worse when relevant information is placed in the middle of the context compared to the beginning or end. For large repositories, no context window is large enough to include everything relevant, making retrieval quality a binding constraint on completion quality.

Latency-quality tradeoff. Larger models produce better completions but have higher latency. The operating point must be chosen based on the task: line-level completions need sub-200ms response, justifying a smaller, faster model. Longer generation tasks (test generation, documentation) can tolerate more latency and benefit from a larger model. Systems often use model routing, directing short completions to smaller models and triggering the larger model only for explicitly requested longer generations.

Diversity and exploration. If the completion system always shows the single highest-probability suggestion, it learns only from that suggestion's accept/reject signal. Occasionally showing diverse alternatives (top-3 suggestions in a menu, or switching between models) provides richer training signal and may surface more useful suggestions for less common patterns. The tradeoff is increased visual complexity and the risk of overwhelming developers with options.

Security and trust. Developers who habitually accept completions without reading them carefully may inadvertently introduce vulnerabilities. Insecure coding patterns are common in open-source training data, and models that memorize these patterns will suggest them. A model trained on millions of lines of code that include SQL string concatenation, unvalidated inputs, or hardcoded credentials will reproduce those patterns. Systems that integrate static analysis to flag suggested code with known vulnerability patterns can add a safety layer, but this increases latency and may introduce false positives that undermine trust.

Evaluation is hard. Acceptance rate is an easy metric to measure but a noisy proxy for actual developer productivity. A completion might be accepted because it is correct, or because the developer is distracted and accepting without fully reading. Downstream metrics like time-to-completion, number of edits after acceptance, and bug rates in accepted-completion code provide a more complete picture but are harder to collect and attribute. The gold standard would be a randomized controlled trial comparing developer productivity with and without the completion system, but such experiments are logistically challenging and ethically complicated (asking some developers to deliberately use a worse tool).

License and copyright concerns. Models trained on open-source code may reproduce verbatim snippets from their training data, creating potential license compliance issues. Some organizations have policies against using LLM-generated code for exactly this reason. Detection systems that identify when a completion closely matches training data and add appropriate attribution (or avoid showing the completion entirely) are an active area of development.

Code completion bridges the gap between what a developer has typed and what they intend to write. The key insights from this chapter are:

• Context construction is an active design problem. The right context goes beyond raw file text. It is a curated combination of prefix, suffix, and retrieved cross-file snippets assembled within a token budget, with the prefix receiving the largest share.
• Fill-in-the-middle (FIM) training enables completion anywhere in a file, not just at the end, by teaching the model to predict masked spans given both prefix and suffix context. This is critical for insertion tasks, which are a large fraction of real developer edits.
• Ranking uses length-normalized log-probability as its primary signal, supplemented by post-processing filters for syntax validity and identifier existence, and optionally a reranker fine-tuned on acceptance data.
• Latency has a hard developer tolerance threshold of roughly 300-500ms for inline suggestions. Speculative decoding, KV cache prefix sharing, debouncing, and streaming delivery are the main tools for meeting this threshold.
• Ghost text is the dominant UX pattern for inline completions because it is non-disruptive: a single keystroke to accept, no action required to dismiss. The asymmetry between accept cost (low) and dismiss cost (near zero) drives adoption.
• Acceptance rate varies substantially by language, task type, and developer experience, and is only a proxy for the true goal of improving developer productivity.
• Systematic failure modes include hallucinated identifiers, lost-in-the-middle attention degradation, security risks from insecure training patterns, and evaluation challenges that make it difficult to measure true productivity impact.

The next chapter examines code generation at a larger scale, where the model produces entire functions or modules from natural language descriptions rather than completing already-started code.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about code completion.