Code LLM Training › The Historical Arc of Code Models

Understanding how code models evolved helps you see why current techniques exist. Each generation of models identified a specific limitation in the previous one and introduced a targeted fix. The progression is not linear, but there is a clear pattern: models got better by solving concrete problems rather than by scaling up existing approaches.

Before neural code models, the dominant paradigm was static analysis and rule-based completion. Integrated development environments like Eclipse and IntelliJ provided code completion by analyzing the abstract syntax tree (AST) of the file being edited and using type information to filter suggestions. These systems were precise, in the sense that every suggestion was syntactically valid and type-correct, but they were brittle. They could not suggest a function name that fit the context, fill in a complex expression, or understand what the developer was trying to accomplish from incomplete information.

Statistical language models were applied to code as early as the mid-2000s. Naturalness of software, a concept developed by Devanbu and colleagues at UC Davis, proposed that code is repetitive and predictable in ways similar to natural language, and that n-gram models trained on source code could predict the next token with surprising accuracy. These n-gram models served as the theoretical foundation for later neural approaches by establishing that code follows statistical regularities that machine learning can exploit.

The first wave of neural code models applied standard recurrent architectures (LSTMs and GRUs) to source code. Models like code2vec represented code as paths through ASTs and learned embeddings that captured structural properties of functions. These models were useful for specific tasks like clone detection and method name prediction but were not general code generators.

The transformer era changed this. GitHub released Copilot in 2021, based on the Codex model, which demonstrated for the first time that a large autoregressive transformer trained on public code could generate useful, complex functions from natural language descriptions. Codex was trained on roughly 100 billion tokens of code from GitHub and showed that the same architecture and training recipe that worked for natural language scaled naturally to code.

The key insight from Codex was that code generation is fundamentally a language modeling problem. You do not need specialized architectures or code-specific inductive biases in the model. What you need is sufficient scale, high-quality code data, and a tokenizer that handles code well.

The second wave introduced structure-aware objectives and bidirectional understanding. GraphCodeBERT added data flow graphs as additional input structure. CodeBERT used bimodal training with natural language-code pairs. These models showed that incorporating program semantics beyond token sequences improves performance on code understanding tasks.

This generation also introduced the idea that code and natural language are not separate modalities but complementary views of the same underlying intent. A function and its docstring describe the same thing in different languages. Training a model to understand both simultaneously, and to translate between them, produces representations that are more useful than either alone.

The third wave, represented by StarCoder, CodeLlama, and Qwen-Coder, addressed the completion-only limitation of autoregressive models by introducing FIM and expanded the context window to include repository-level information. These models could infill arbitrary code spans and attend to related files, bringing code model capability closer to what a developer experiences when working in a large codebase.

This generation also significantly increased scale: StarCoder was trained on 1 trillion tokens, CodeLlama used 500 billion code-specific tokens for a continued pretraining stage on top of Llama-2, and DeepSeek Coder trained from scratch on 2 trillion tokens. The scale increase, combined with improved data curation, produced models that substantially outperformed the first generation on all benchmarks.

The current state of code models combines large-scale pretraining with instruction tuning and alignment. Models like GPT-4 and Claude are not code-specific, but they achieve strong coding performance because their general capability includes code comprehension and generation as special cases. Code-specific models like DeepSeek Coder V2 and Qwen2.5-Coder achieve competitive performance on coding benchmarks while being more efficient than general-purpose models of similar size.

The trend suggests that the boundary between code models and general language models is dissolving. A model that understands natural language well also understands the natural language in comments, docstrings, and commit messages, which helps it understand code. A model that understands code well can reason about structured data formats like JSON and XML, which helps it handle a wider range of tasks.