Code Completion › Completion Context

The quality of any completion model is bounded by the quality of the context it receives. Feed the model too little context and it cannot produce relevant suggestions. Feed it too much and you either exceed the model's context window, add unnecessary latency, or dilute the signal with irrelevant text. Constructing the right context is therefore one of the most consequential decisions in the entire system.

Code completion context is not simply "everything before the cursor." It is a carefully engineered view of the codebase, assembled from multiple sources and filtered for relevance. The assembler must balance recency (the most recently written code is the strongest signal), scope (the surrounding function, class, and module matter more than distant code), and diversity (sometimes an example from a completely different file is the key piece of context the model needs).

Early code completion systems only considered the text before the cursor, called the prefix. The model was asked to continue from where the developer stopped. This works when the developer is writing new code at the end of a function, but it fails badly when they are inserting code in the middle of an existing block or filling in an argument to a function call.

Consider a developer who has already written the outer structure of a function and now wants to fill in its body. With a prefix-only model, the model sees the beginning of the function but not the return statement that constrains what the body must produce. This leads to completions that are locally plausible but globally wrong: the body might manipulate the wrong variables or fail to produce a value matching the type the caller expects.

Modern systems solve this with a training objective called fill-in-the-middle (FIM). Instead of training the model only to predict continuations, FIM training masks a random span from the middle of a document and asks the model to reconstruct the missing portion given both the prefix (text before the span) and the suffix (text after the span). At inference time, the editor sends the prefix and suffix separately, and the model produces the infill that fits between them.

The FIM format rearranges the input into a structure like:

where <PRE>, <SUF>, and <MID> are special tokens the model learns to associate with their respective roles. The model then generates tokens that would naturally connect the prefix to the suffix. This matters enormously in practice because a large fraction of real completions are insertions: adding a parameter to an existing function call, writing the body of a function whose signature already exists, or filling in a missing branch of a conditional.

FIM training also changes what the model learns about code structure. With prefix-only training, the model develops a strong prior toward producing code that looks syntactically complete: it always writes closing braces, return statements, and function endings. With FIM training, the model learns to produce partial fragments that are syntactically open-ended, trusting that the suffix provides the closing structure. This seemingly small change substantially improves completion quality for insertion tasks, which in practice account for a large share of developer edits.

Research on models like InCoder and CodeLlama showed that FIM-trained models achieve significantly higher exact-match rates on insertion benchmarks than comparable prefix-only models, with the gap widening for longer insertions. The reason is intuitive: for a short one-token insertion the suffix provides little additional information, but for a multi-line insertion the suffix is an extremely strong constraint that eliminates entire classes of plausible-but-wrong completions.

The simplest completion context is the contents of the file currently open in the editor. For short files, the entire file fits in the model's context window and provides useful signals: the imports at the top tell the model what libraries are in use, the function signatures already defined tell it what API the developer is building, and the variable names in scope tell it what identifiers are available.

Short files are the easy case. The challenge arises for long files, where the developer has accumulated hundreds or thousands of lines across multiple classes and functions. Such files routinely exceed the context windows of smaller completion models (typically 2K to 8K tokens for fast inference), and even modern models with 128K context windows face a practical tradeoff: longer contexts mean higher prefill latency.

For long files that exceed the context window, the system must decide which portions to include. A common strategy is to always include the full prefix up to the cursor (since the most recent code is the strongest signal), then fill remaining context tokens with the suffix starting from the cursor, then optionally include earlier sections of the file if budget remains. This truncation heuristic encodes a model of relevance. The code immediately around the cursor is more relevant than code far away, the current function is more relevant than unrelated functions, and recently modified code is more relevant than stable legacy code.

An alternative approach, used in some production systems, is to build a per-file index that identifies the most relevant fragments to include even if they are far from the cursor. For example, if the cursor is inside a method that calls several helper functions defined earlier in the same file, including those helper function signatures (even if their bodies are omitted) provides the model with the type and interface information it needs without consuming the entire context budget.

A single file is rarely sufficient. Modern codebases are modular: a developer writing a new function might need context from type definitions in one file, utility functions in another, and interface contracts in a third. Without cross-file context, the model cannot see the types of the arguments it is supposed to use, the methods available on the objects it is working with, or the conventions established by adjacent code.

Cross-file context requires a retrieval step. The system builds an index of the workspace (often a vector store or a BM25-style inverted index of identifiers and snippets) and queries it at completion time with the local context as a probe. The retrieved snippets are then prepended to the completion request as additional context.

Several retrieval signals are useful:

• Import graph proximity: Files imported by the current file are likely more relevant than files elsewhere in the project. If the current file imports auth.utils, then the contents of auth/utils.py are immediately relevant.
• Identifier overlap: Files that define identifiers referenced in the current file are highly relevant. If the cursor is inside a call to UserRepository, the file that defines UserRepository should be in context.
• Recency: Files the developer has recently edited or viewed are often related to the current task. Most developers work on a small set of files at a time, and recently opened files are likely thematically related.
• Semantic similarity: Embedding-based similarity can surface relevant code that shares no common identifiers but solves a similar problem. This is particularly valuable for surfacing patterns from elsewhere in the codebase that the developer might want to follow.

The challenge is that retrieving and embedding snippets takes time. Production systems use aggressive caching: index updates happen in the background as files change, and common queries are cached for a short window so that nearby keystrokes reuse the same retrieved context rather than triggering fresh retrieval. The index itself is often a combination of a fast lexical index for identifier lookup and a slower but more powerful vector index for semantic similarity.

The architecture of cross-file retrieval resembles the retrieval-augmented generation (RAG) pipelines we examined earlier in this book, adapted for code. The key differences are that the documents (code files) have strong structural constraints (syntax, scope, module boundaries), the queries (the local context around the cursor) are code rather than natural language, and latency requirements are far stricter than in typical question-answering RAG systems.

Beyond individual files, some completion systems attempt to understand the broader structure of a repository: its conventions for naming, its test patterns, its error handling idioms, and its architectural boundaries. This is sometimes called repository-level context or long-context completion.

One approach encodes a summary of the repository into a persistent context that is prepended to every completion request. The summary might include the repository's main modules, common patterns extracted from the codebase, and project-level configuration (such as which version of a library is being used, which could affect API availability). Another approach uses a retrieval-augmented generation (RAG) setup with chunks of the repository indexed by a code-aware embedding model.

Repository-level context is particularly valuable for large, mature codebases where the developer is expected to follow established patterns. A model that knows the repository uses a specific logging framework, a specific error-handling convention, and a specific style for writing tests will produce far more useful suggestions than one that relies only on general training data. This is one reason that enterprise code completion systems often outperform general-purpose completions for developers working in large internal codebases.

The practical limit here is latency. Encoding long contexts takes time, and retrieval introduces its own delay. Systems that make aggressive use of repository context typically do so in a background prefetch mode, building up a cached representation that is ready when the developer triggers a completion. The cached representation is invalidated when files change, triggering a background refresh that completes before the developer's next completion request.

Given a fixed context window size (say, 8192 tokens for a smaller model or 128K for a larger one), how should the budget be allocated across prefix, suffix, and retrieved snippets?

The allocation problem optimizes the expected completion quality given the model's training distribution and the typical distribution of completion requests. A model that was trained primarily on single-file contexts will benefit less from cross-file snippets than one specifically trained with repository context. A model with a large context window can afford to be generous with retrieved snippets, while a small model must be much more selective.

A typical allocation strategy looks like this:

• Prefix: Allocate the largest share, often 50-70% of the budget. The most recent code is the strongest signal.
• Suffix: Allocate 15-25% of the budget. Useful for FIM but less critical than prefix.
• Retrieved snippets: Allocate the remainder. Sort retrieved snippets by relevance score and include as many as fit.

This is not a universal formula. The right allocation depends on the model's training distribution, the average file length in the target language, and empirical evaluation on completion benchmarks. Systems that have been deployed at scale typically discover through A/B testing that their initial allocation assumptions were suboptimal, and iterate toward data-driven allocation policies.