Code Completion › Summary

Code completion bridges the gap between what a developer has typed and what they intend to write. The key insights from this chapter are:

• Context construction is an active design problem. The right context goes beyond raw file text. It is a curated combination of prefix, suffix, and retrieved cross-file snippets assembled within a token budget, with the prefix receiving the largest share.
• Fill-in-the-middle (FIM) training enables completion anywhere in a file, not just at the end, by teaching the model to predict masked spans given both prefix and suffix context. This is critical for insertion tasks, which are a large fraction of real developer edits.
• Ranking uses length-normalized log-probability as its primary signal, supplemented by post-processing filters for syntax validity and identifier existence, and optionally a reranker fine-tuned on acceptance data.
• Latency has a hard developer tolerance threshold of roughly 300-500ms for inline suggestions. Speculative decoding, KV cache prefix sharing, debouncing, and streaming delivery are the main tools for meeting this threshold.
• Ghost text is the dominant UX pattern for inline completions because it is non-disruptive: a single keystroke to accept, no action required to dismiss. The asymmetry between accept cost (low) and dismiss cost (near zero) drives adoption.
• Acceptance rate varies substantially by language, task type, and developer experience, and is only a proxy for the true goal of improving developer productivity.
• Systematic failure modes include hallucinated identifiers, lost-in-the-middle attention degradation, security risks from insecure training patterns, and evaluation challenges that make it difficult to measure true productivity impact.

The next chapter examines code generation at a larger scale, where the model produces entire functions or modules from natural language descriptions rather than completing already-started code.