Code Completion › Completion Ranking

Given a completion context, the model produces a probability distribution over possible next tokens at each step. When using a model that generates completions autoregressively, the full sequence is decoded and returned as a suggestion. But often, the system generates multiple candidate completions and must decide which one to show the developer first.

Ranking is the problem of ordering candidate completions by their expected quality. A completion that is syntactically correct, semantically appropriate, and matches the developer's intent should rank above one that is merely probable under the model. This distinction matters because the model's probability distribution reflects patterns in training data, not the developer's actual intent in the current moment. A highly probable completion might be a common boilerplate that is wrong for this specific use case, while a slightly less probable completion might be exactly what the developer needs.

The simplest ranking signal is the model's own probability score for the generated sequence. For a completion $c = (c_1, c_2, \ldots, c_T)$, the log-probability is:

$$\log P(c) = \sum_{t=1}^{T} \log P(c_t \mid c_1, \ldots, c_{t-1}, \text{context})$$

where:

• $c_t$: the $t$-th token in the completion
• $T$: the length of the completion in tokens
• $\text{context}$: the prefix, suffix, and any retrieved snippets
• $\log P(c_t \mid \cdots)$: the model's log-probability for token $c_t$ given all preceding context

The raw log-probability sums across all tokens, which means longer completions are systematically penalized: a completion of 20 tokens will almost always have a lower total log-probability than a 5-token completion, even if every token in the 20-token completion is equally probable. This makes length-normalized comparison essential.

Longer completions tend to have lower log-probabilities simply because they contain more tokens, each of which contributes a multiplicative penalty. To compare completions of different lengths fairly, systems typically use the mean log-probability:

$$\text{score}(c) = \frac{1}{T} \sum_{t=1}^{T} \log P(c_t \mid c_1, \ldots, c_{t-1}, \text{context})$$

This length-normalized score penalizes completions with many low-probability tokens relative to completions where every token is highly confident. A completion where every token has mean log-probability of -0.2 is more confident than one where some tokens have log-probability -2.0, even if the overall sequence length is the same.

Log-probability has important limitations as a ranking signal. The model's probability distribution is calibrated to its training data distribution, not to the developer's immediate intent. A completion that is common in training data (and therefore high-probability) might be exactly what we do not want here. For example, if the model has seen many Python files that use assert x > 0 for input validation, it might assign high probability to that pattern even when the developer is writing production code where exceptions are more appropriate. Log-probability captures "how typical is this" rather than "how correct is this for this particular situation."

Before ranking, candidates are filtered for basic validity. A completion that fails to parse as valid syntax in the target language is almost certainly wrong, even if the model assigned it a high probability. Common filters include:

• Syntax validity: Parse the completion and reject completions with syntax errors. This is fast in most languages using incremental parsers. Tree-sitter, a popular incremental parsing library used in many editors, can parse partial code fragments and identify syntax errors in under a millisecond.
• Type consistency: In statically typed languages, check that the completion does not introduce obvious type errors. This is harder and typically requires integration with the language server. Systems that have access to a running language server can query it for type information at low latency.
• Identifier existence: Check that identifiers referenced in the completion are in scope. Hallucinated method names or nonexistent variables are a common failure mode, particularly for APIs the model has not seen frequently in training.
• Style matching: Strip completions that violate project-level formatting rules (trailing whitespace, incorrect indentation style). While these are cosmetic, accepting a completion that immediately triggers a linting error creates a negative experience.

These filters reduce the candidate set before expensive reranking, improving both quality and latency. The key insight is that syntactic filters are fast (sub-millisecond) and catch a disproportionate fraction of bad completions, while type-based filters are slower but catch more subtle errors.

The primary completion model is often large, making it expensive to generate many diverse candidates. A common pipeline uses the large model to generate a small number of candidates (say, three to five), then uses a separate smaller reranker model to score them and reorder the list.

The reranker is typically a sequence classifier or a smaller language model fine-tuned to predict acceptance probability. It receives the context and a candidate completion as input and outputs a scalar score. Because the reranker only scores rather than generates, it is much faster than the generation model and can be run in parallel on all candidates.

Fine-tuning the reranker on real acceptance data (completions accepted by developers versus dismissed ones) grounds the scoring in observable human behavior rather than proxy metrics like log-probability. A developer who types the exact same code that a dismissed completion suggested is providing a clear signal: the suggestion was correct, but perhaps it appeared at the wrong moment or was visually presented in a way that caused the developer to miss it. A developer who accepts a completion and immediately backspaces is providing the opposite signal. These timing and correction signals can be learned by a reranker trained on large logs of developer interactions.

One important consideration in reranker design is the distribution shift between training and deployment. Rerankers trained on historical acceptance data inherit any biases in what completions were shown historically. If the system previously favored certain completion styles, the reranker will have more training signal for those styles and may unfairly downrank novel completion types that were rarely shown. Mitigating this requires deliberate data diversity in training or explicit correction for historical presentation bias.

When a developer accepts a completion, that is a strong positive signal. When they dismiss it or immediately type over it, that is a signal it was not useful. Collecting these acceptance and rejection events and using them to fine-tune both the generation model and the reranker closes the feedback loop between the deployed system and real developer behavior.

This is an instance of reinforcement learning from human feedback (RLHF) applied at the system level rather than the conversation level. The acceptance event plays the role of the human preference signal, and the completion model is updated to produce more accepted suggestions over time. The key difference from conversational RLHF is that the feedback is implicit (the developer does not rate the suggestion) and abundant (millions of completion events per day in large deployments).

The practical challenge is selection bias: developers only see the top-ranked completion, so you observe acceptance or rejection of that completion but not of alternatives. If you knew the developer would have accepted completion B but you showed them completion A, you never learn that signal. Systems that occasionally surface lower-ranked completions (with deliberate exploration) can gather richer training data, though this must be balanced against user experience. Exploring too frequently degrades the product quality that real users experience; exploring too rarely leads to a model that is overfit to the historical top-ranked distribution.

Some systems address this by running A/B experiments at the user level rather than the suggestion level: a randomly selected subset of users receives an experimental ranking model, and their aggregate acceptance rates provide a signal about which ranking strategy performs better without degrading every individual's experience.