Code LLM Training: Tokenization, FIM, Pretraining Objectives

Code-generating language models do not simply learn to imitate programming syntax. They learn the reasoning structure of software: how functions are composed, how control flow encodes intent, how identifier names carry semantic weight, and how a snippet of code relates to surrounding context in ways that matter for correctness. Getting a model to reliably generate, complete, and explain code requires rethinking nearly every stage of the training pipeline, from the raw bytes of source files all the way through the optimization objective.

This chapter traces that pipeline in full. You will see why code data requires different curation than natural language, why tokenizers for code need to be designed with identifier structure and indentation in mind, what fill-in-the-middle training is and why it transforms what completion models can do, and what pre-training objectives have proven most effective for code models. By the end, you will understand the mechanics of code LLM training and why each design decision exists.

The quality and composition of training data is the single most consequential decision in building a code model. Unlike natural language pretraining, where any sufficiently large corpus of web text provides useful signal, code data requires deliberate filtering, deduplication, and structural understanding. The difference matters because code has properties that web text does not: it can be syntactically correct yet semantically wrong, it exists in enormous quantities of low-quality form (generated, copied, or trivially repetitive), and its quality can sometimes be verified automatically by running it.

The primary source of code training data is public version control hosting. GitHub, GitLab, and similar platforms host hundreds of millions of repositories containing code in thousands of programming languages. The GitHub Code dataset, used in early Codex training, contained roughly 54 million public repositories. The Stack, assembled by BigCode, covers over 350 programming languages and reached approximately 3 terabytes of source code after initial filtering.

Beyond repositories, code data comes from several complementary sources, each offering a different kind of learning signal:

• Documentation and tutorials: Official language documentation, framework guides, and tutorials contain correct, idiomatic code accompanied by natural language explanations. This is especially valuable because it pairs code with intent, which is precisely the kind of alignment a code assistant needs. A model trained on documentation learns how functions are implemented and what they are for.
• Competitive programming: Platforms like Codeforces, LeetCode, and HackerRank provide algorithmically challenging problems with multiple correct solutions, often accompanied by comments explaining the approach. This data pushes the model to learn algorithmic reasoning rather than only rote code patterns.
• Notebooks: Jupyter notebooks from GitHub and Kaggle provide code interleaved with markdown explanations, giving fine-grained alignment between prose and code at the cell level. Models trained on notebook data tend to produce better-explained code and are more capable of describing what a block of code does.
• Commit histories: Diffs between commits capture how code evolves, which can be used to train models on code editing and bug fixing. A commit that changes a function from an incorrect to a correct version is a golden example of the editing task that real developers encounter.
• Issues and pull requests: Discussion threads on GitHub pair natural language problem descriptions with code patches, enabling instruction-following for code tasks. These are especially valuable for fine-tuning, since they closely resemble the conversational format in which developers interact with code assistants.

The ratio between these sources matters. Models trained primarily on raw source files learn syntax and structure but may struggle with natural language descriptions of problems. Models with more documentation and notebook data tend to generalize better to instruction-following and code explanation tasks. The early Codex model was criticized for being excellent at generating plausible code but inconsistent at following docstring-based specifications, a limitation attributed in part to the dominance of raw source files in its training mix.

Programming languages vary enormously in how much public code exists. Python, JavaScript, Java, and C/C++ dominate public repositories by several orders of magnitude. Languages like Rust, Haskell, or niche domain-specific languages have far less public code available. Languages like Fortran, COBOL, or assembly exist in even smaller quantities.

Naive training on the raw distribution would produce a model that is excellent at Python and JavaScript but nearly useless for Rust, Go, or TypeScript. This is a serious problem for tools marketed as general-purpose code assistants, since developers using those languages need help just as much as Python developers do.

Two strategies address this imbalance:

Upsampling rare languages: Multiply the effective frequency of underrepresented languages during training. If Rust represents 0.1% of the raw corpus but you want similar performance to Python, you sample Rust files at 10 to 50 times their natural frequency. The tradeoff is that upsampling can cause overfitting on the smaller Rust corpus, since the model sees the same files repeatedly. A Rust file might be presented hundreds of times across training while a Python file appears only once.

Minimum language thresholds: Set a minimum token count per language below which the language is excluded entirely. This prevents the model from learning partial, low-quality representations of languages with only a few hundred files, which are worse than learning nothing at all. A few hundred Fortran files will teach the model some Fortran syntax but may also introduce subtle errors or non-idiomatic patterns that are hard to detect.

The Stack v2 and similar recent datasets take a principled approach, targeting rough parity between languages with substantial public code while excluding languages with fewer than a few thousand repositories. Research from BigCode found that language balance significantly affects downstream benchmark performance on multilingual code evaluation. Interestingly, cross-language transfer also matters: a model with strong Python understanding can generalize better to TypeScript or Kotlin than a model trained on fewer total languages, even if the latter was trained on more examples of each specific language. This transfer effect suggests that learning the common structure of programs across languages provides a useful inductive bias.

Raw repository data contains enormous amounts of noise. A repository scraped from GitHub might include auto-generated code from build systems or protobuf compilers, minified JavaScript where all whitespace and identifiers have been stripped, binary files with .py or .js extensions, duplicate code copied between repositories with minor modifications, low-quality educational files with repetitive trivial content, and files that are primarily comments with almost no actual implementation.

Each of these adds to training time and data storage while providing minimal learning signal. More dangerously, some kinds of noise actively degrade model quality. Minified code teaches the model that long single-line statements without whitespace are valid Python. Auto-generated boilerplate teaches the model that repetitive patterns with no semantic meaning are normal. These biases show up as quirks in generated code that are difficult to remove after the fact.

Standard filtering steps include several complementary approaches:

Exact deduplication: Remove identical files across repositories. Surprisingly, a large fraction of public code is direct copies. The Stack found that approximately 70% of files could be removed through near-deduplication, leaving only the highest-quality representative copy. The sheer scale of this duplication reflects how common it is for developers to copy-paste starter templates, configuration files, or example implementations.

Near-deduplication using MinHash: Exact deduplication misses slightly modified copies. MinHash locality-sensitive hashing identifies files with above a certain Jaccard similarity threshold, typically 0.8 or higher, and keeps only one representative. This dramatically reduces memorization of boilerplate. Without near-deduplication, a model may memorize common license headers, README templates, or configuration snippets at the expense of learning more diverse code patterns.

Quality heuristics: Filter files by metrics including line length distribution (very long average lines indicate minified code), alphabetic character fraction (very low fraction indicates binary or generated content), number of unique lines as a fraction of total lines (very low fraction indicates repetitive files), and comment-to-code ratio (files with no code lines are documentation, not implementation). These heuristics are fast to compute at scale and remove the most obvious garbage.

License filtering: Many models filter to permissively licensed code such as MIT, Apache 2.0, and BSD to reduce legal risk. The Stack provides license metadata for this purpose. This filtering removes a meaningful fraction of highly useful code, since many projects use restrictive licenses, but it is considered essential for commercial deployment.

One subtle but important aspect of code data is that files rarely exist in isolation. A Python function that calls np.array() is meaningless without knowing that numpy is imported. A TypeScript file uses types defined in adjacent files. A Rust function requires crate dependencies. Understanding code at a file level misses the semantic relationships required for programs to work.

This context problem has direct consequences for how training data is organized. If a model never sees that compute_loss is called with a specific signature in train.py and defined with a matching signature in model.py, it cannot learn the conventions that govern cross-file consistency. It may learn to write plausible function definitions, but it will not learn to write functions that fit naturally into a codebase. The same observation applies to type information: knowing that a function expects an np.ndarray rather than a Python list is essential for generating code that composes correctly, but this information often lives in type stubs or adjacent files.

Early code models like Codex trained on individual files without repository context, which limited their ability to use multi-file context. Later models like GitHub Copilot and StarCoder introduced repository-level context, including adjacent file contents, import graphs, and function signatures from related modules. The AlphaCode dataset organized training data at the problem level, making sure the model saw complete problem specifications alongside solution code.

Repository-level data organization is technically more challenging than file-level organization. It requires crawling the repository's dependency graph, resolving import paths, and deciding how to pack related files together within the context window. Context packing choices matter: including too many irrelevant files dilutes the signal, while including too few misses important cross-file relationships. But the performance gains on real-world coding tasks are substantial enough that repository-level context has become standard practice in production code LLMs.

To make filtering concrete, consider a raw Python file scraped from GitHub. It might be 800 lines long with a one-line copyright header, 700 lines of auto-generated protobuf binding code that looks like class ResponseType_pb2: __slots__ = ['field_a', 'field_b'...] repeated with minor variations, 80 lines of actual logic, and 20 blank lines. The unique-line ratio is approximately 0.12 (most lines are near-copies), so a threshold of 0.4 would reject it. Running it through the alphabetic ratio filter would also likely reject it because of the abundance of brackets and underscores in the generated code.

After filtering, you retain the 80 lines of actual logic, but only if those 80 lines appear in a different file that is not buried inside the auto-generated file. This example illustrates why filtering is effective: it removes the flood of repetitive content while keeping files where most lines are semantically distinct.

Tokenization for code differs from natural language tokenization in ways that go beyond just applying BPE to source text. The structure of identifiers, whitespace sensitivity in indented languages, and the prevalence of long token sequences all affect how tokenization choices translate into model capability. A poorly designed tokenizer does not just produce longer sequences, it actively destroys the semantic structure of code in ways that make learning harder.

Consider the identifier calculateMeanSquaredError. In natural language, this would almost never appear, but splitting it at byte boundaries or random subword boundaries is semantically destructive. The identifier has meaningful internal structure: calculate, Mean, Squared, Error. A tokenizer that splits it as cal|culat|eMean|Squared|Error loses the semantic decomposition that would help a model understand what the function does. Worse, the same identifier split differently at different occurrences in the training data produces inconsistent representations, making the model's job harder.

CamelCase and snake_case identifiers have consistent internal structure that tokenizers should ideally respect. Similarly, compound operators like ->, ::, =>, and ** have unified semantic meanings that should be preserved as single tokens. A model that sees -> split as - and > has a harder time learning that it is a member access operator.

Three properties distinguish effective code tokenizers:

Identifier awareness: The tokenizer should be trained on code data so its vocabulary naturally includes common identifiers, language keywords, and operator sequences as single tokens. A vocabulary trained on web text will not include np.array or torch.nn.Module, while a code-tuned vocabulary may encode these as one or two tokens, dramatically reducing sequence lengths for common patterns.

Whitespace handling: In Python and other indentation-sensitive languages, leading spaces carry syntactic meaning. A tokenizer must represent indentation faithfully. Many code tokenizers reserve special tokens for common indentation patterns such as two spaces, four spaces, eight spaces, and a tab to avoid exploding sequence length in deeply nested code. Without this, a function nested four levels deep might spend a substantial fraction of its token budget just on leading spaces.

Number representation: Source code contains many numeric literals with specific formats. 0x1F3A (hexadecimal), 3.14159 (float), 1e-5 (scientific notation), and 0b1010 (binary) are all valid number literals in most languages. A tokenizer should handle these consistently rather than producing radically different tokenizations for the same numeric value in different notations.

Larger vocabularies reduce sequence length and allow more context to fit within the model's context window for a given amount of source code, but they increase the size of the embedding table and output projection matrix. For a model with hidden dimension $d$ and vocabulary size $V$, the output projection has $V \times d$ parameters. Doubling the vocabulary doubles this component's parameter count. For a model with $d = 4096$ and $V = 100{,}000$, the output projection alone has 409 million parameters, which is larger than some full models.

Small vocabularies (like character-level tokenization) produce very long sequences that are expensive to process, since attention cost scales quadratically with sequence length, but they can represent any code without out-of-vocabulary tokens.

Most production code models use BPE vocabularies in the range of 32,000 to 100,000 tokens, trained specifically on code corpora. The CodeLlama tokenizer uses a vocabulary of 32,000 tokens trained on a code-heavy corpus. StarCoder uses a vocabulary of 49,152 tokens with reserved space for future tokens. GPT-4's tokenizer, which covers both natural language and code, uses approximately 100,000 tokens.

A key tradeoff specific to code is that a vocabulary trained primarily on natural language handles code poorly. Common code patterns like import numpy as np, def __init__(self):, and return None may be split into many tokens when using a natural language tokenizer, while a code-tuned tokenizer would represent each as one or two tokens. CodeLlama's experiments showed that code-specific tokenization reduced effective sequence lengths by roughly 30 to 40 percent compared to using the Llama-2 natural language tokenizer on identical code. This matters because a shorter sequence allows the model to attend to more context with the same memory budget, which directly translates into the model being able to "see" more of the file it is working with.

Most modern code models use byte-level BPE, which begins by treating each byte (0 through 255) as a base token before learning merge operations. This ensures that no input is ever truly out-of-vocabulary: any byte sequence, including unusual Unicode in string literals, emoji in comments, or non-ASCII identifier characters in international codebases, can be represented.

Byte-level tokenization is especially important for code because source files regularly contain string literals with arbitrary Unicode content, regular expressions with unusual character classes, code comments in languages other than English, and hex-encoded data or base64 constants embedded in source. Without byte-level coverage, these would require special handling or would produce unknown tokens that degrade model quality.

The BPE algorithm then learns merges from this byte base, starting with the most frequent pairs of bytes and iteratively combining them into longer tokens. By training on code-specific corpora, the learned merges naturally favor code patterns: common function names, operator sequences, and indentation patterns become single tokens through repeated merging, while less common byte sequences remain as multi-byte representations.

Python is arguably the hardest language to tokenize well, because its syntax depends on precise indentation to define block structure. A function body indented by four spaces and one indented by three spaces are syntactically different programs (the latter is a syntax error). If indentation is tokenized inconsistently, the model must learn to count spaces character by character, which is wasteful and error-prone.

One solution is to include dedented tokens in the vocabulary: a token that represents four spaces, another for eight spaces, another for twelve spaces, and so on. Quoting the SantaCoder paper, using dedicated indentation tokens reduces Python sequence lengths by approximately 15 percent compared to using the standard GPT-2 tokenizer with no special indentation handling. This is a meaningful gain given Python's dominance in code model training data.

The indentation problem also interacts with FIM. When a FIM split falls at a line boundary, the first token of the middle region is often an indentation sequence followed by code. If the tokenizer does not handle indentation efficiently, the beginning of each infill is dominated by whitespace tokens, making it harder for the model to produce the first meaningful token quickly. In practice, this is why models trained with FIM often include indentation-aware tokenization as a paired improvement.

Numeric literals deserve special attention because they appear in code with high frequency and in diverse formats. Consider the range of ways a value like 255 can be written in Python: 255, 0xFF, 0b11111111, 0o377. These are all the same integer, but their character representations are completely different. A tokenizer that handles them inconsistently may learn that 0xFF and 255 are unrelated values.

Most modern code tokenizers handle this pragmatically by including common hexadecimal, binary, and octal patterns as high-frequency merge targets in BPE, so that 0xFF becomes a small number of tokens while preserving the prefix structure. A more principled approach would canonicalize numeric literals during preprocessing, converting all integers to decimal before tokenization. This is used in some research settings but is not standard in production models because it changes the surface form of the code, which may cause issues when the model needs to reconstruct the original representation.

Floating-point literals present a different challenge: 3.14159, 3.14159e0, and 3.14159E+00 are the same value but tokenize differently. Scientific notation in particular is common in machine learning code (learning rates, epsilon values) and numerical computing. A tokenizer trained on sufficient code data will typically learn that 1e-5 is a natural unit and encode it efficiently.

The most significant training innovation specific to code models is fill-in-the-middle (FIM) training. It fundamentally changes what completion models can do and is worth understanding in depth, both in terms of the mechanics of how it works and the intuition for why it is so effective.

Standard autoregressive language models predict the next token given all previous tokens. This makes them excellent at completing code from a prefix: given the beginning of a function, they continue it forward. But software developers spend a large fraction of their time not completing code forward from a cursor, but rather inserting code in the middle of existing code.

Consider these scenarios that require infilling:

• Writing the body of a function when the signature and docstring are already present
• Filling in a missing function between two complete ones
• Implementing a try block when the except clause is already written
• Completing a line in the middle of a larger codebase where surrounding context is already established
• Filling in argument lists where the function call structure is already partially written

A pure left-to-right model cannot use the suffix (code after the cursor) as context. It is forced to treat completion as if nothing comes after, which produces code that may not connect correctly with what follows. A model generating a function body without knowing what comes after it may choose variable names or return types that are incompatible with the calling code that already exists below the cursor.

This is not a minor usability issue. For an IDE completion assistant, infilling is arguably the dominant use case. Developers rarely write code from scratch in a single forward pass; they more commonly add a function somewhere in the middle of a file, fill in an argument to a function call, or implement a method in a class that already has several methods defined.

Fill-in-the-middle training teaches the model to use both prefix and suffix context when generating the infill region. The key insight is that this capability can be trained into a standard autoregressive model without modifying its architecture, simply by rearranging the order in which tokens are presented during training.

FIM works by transforming the document order of training examples. Given a complete source document, three special tokens are introduced:

• <PRE> (prefix sentinel): marks the start of the prefix context
• <SUF> (suffix sentinel): marks the start of the suffix context
• <MID> (middle/infill sentinel): marks where the model should begin generating the middle content

A random split point is chosen within the document, dividing it into prefix, middle, and suffix portions. The document order is then rearranged to present the task as: given prefix and suffix, predict middle. The transformed document looks like:

The model is trained autoregressively on this rearranged sequence. By the time the model reaches <MID>, it has attended to both the prefix (before the split point) and the suffix (after the split point) during the forward pass. The middle tokens are then predicted causally given this combined context.

At inference time, the model receives <PRE> {prefix} <SUF> {suffix} <MID> and must generate the middle portion. This is the same format as training, so the model knows exactly what is expected of it.

There are two main variants of the FIM transformation, originally described in the Starcoder paper and related work:

PSM (Prefix-Suffix-Middle): The arrangement shown above. The model sees the prefix first, then the suffix, then generates the middle. PSM has the advantage that the prefix is processed first and may set up important context before the suffix is read.

SPM (Suffix-Prefix-Middle): The suffix is presented first, followed by the prefix, and then the model generates the middle. In SPM format, the document looks like:

SPM was found to be more effective in some evaluations because it allows the model to immediately attend to the suffix without having first processed a potentially long and distracting prefix. When the suffix is short and highly constraining (for example, return result), presenting it first allows the model to condition everything that follows on this constraint.

In practice, most code models train on a mixture of SPM and PSM examples, with FIM applying to approximately 50 percent of training examples. The remaining 50 percent are trained in standard left-to-right order to preserve the model's completion capability from prefix-only contexts. This mixed training ensures the model does not lose generative quality in the standard completion scenario, which is still common.

At first glance, FIM might seem like a hack: you are presenting tokens out of their natural order and asking the model to treat it as if this were normal. But the mathematical reason FIM works is straightforward. A standard autoregressive model learns the distribution:

By reordering the document as <PRE> prefix <SUF> suffix <MID> middle, FIM trains the model to learn a conditional:

This conditional distribution is exactly what you want for infilling. The model learns that when it sees <PRE>...<SUF>...<MID>, the correct action is to generate content that bridges the prefix and suffix. Because the reordered document is still a valid token sequence, the standard autoregressive training loop applies without modification. No architectural changes are needed.

The softness here is that the model does not have true bidirectional attention to the suffix. When generating middle token $t_m$, the model has attended to the suffix in its prefix context, but it cannot attend to the middle tokens it has already generated in relation to the suffix. This contrasts with masked language models, which use bidirectional attention. In practice, this limitation is largely mitigated by the model learning to "plan" the infill based on the suffix it has already read.

The original FIM formulation splits at a randomly chosen character position, which creates infill spans of arbitrary size. In practice, the training distribution of span types matters for downstream performance:

Character-level splits: Split at a uniformly random character position. This creates a wide variety of span lengths, from single characters to most of the document. Character-level splits are maximally diverse but produce many awkward examples where the split falls inside a token (for example, splitting calculateMean at the c after cal).

Line-level splits: Split at line boundaries. This is better aligned with how code editors implement completion, where the cursor typically sits at the end of a line or between lines. Line-level splits produce more realistic training examples and are used in most production code models.

Identifier-level splits: Split at identifier or token boundaries so that the prefix and suffix are syntactically complete. This avoids training the model on examples where splits cut through tokens, which are unrealistic completion scenarios.

CodeLlama used a combination of line-level and character-level splitting, finding that including some character-level examples improved robustness to mid-line cursor positions common in IDEs. The intuition is that a model trained only on line-level splits will not have seen the task of completing a partially written identifier, a common situation in real usage.

A significant enhancement to basic FIM is providing repository context. When filling in a function body, the most useful context is often not the immediately surrounding lines but the signatures of related functions in the same file, the types defined in imported modules, or the docstrings of functions that the infill will call.

Repository-level FIM constructs the prefix and suffix from the current file and from a retrieval-augmented view of the repository. A retrieval system identifies the most relevant adjacent files based on import relationships, semantic similarity, or co-occurrence in the version history, and their contents are prepended to the context window.

This was a core innovation in models like Copilot Chat, where the IDE extension retrieves semantically relevant code from the open project and includes it in the context before calling the model. The retrieval is done client-side in the IDE, so the model itself does not need to learn retrieval, it just needs to use the retrieved context effectively.

To make FIM concrete, let's trace through a specific example. Suppose the training document is:

A line-level FIM split at line 3 (zero-indexed) produces:

• Prefix: def compute_area(radius):\n """Compute the area of a circle."""
• Middle: area = 3.14159 * radius ** 2
• Suffix: return area\n\nresult = compute_area(5.0)\nprint(f"Area: {result:.2f}")

The PSM-transformed training example becomes:

The model is trained to predict the middle line area = 3.14159 * radius ** 2 given the function signature, docstring, and the return statement plus caller. At inference time, a developer who has already written the function signature, the docstring, and the return statement would receive exactly this infill as a suggestion, which is both natural and useful.

Understanding how code models evolved helps you see why current techniques exist. Each generation of models identified a specific limitation in the previous one and introduced a targeted fix. The progression is not linear, but there is a clear pattern: models got better by solving concrete problems rather than by scaling up existing approaches.

Before neural code models, the dominant paradigm was static analysis and rule-based completion. Integrated development environments like Eclipse and IntelliJ provided code completion by analyzing the abstract syntax tree (AST) of the file being edited and using type information to filter suggestions. These systems were precise, in the sense that every suggestion was syntactically valid and type-correct, but they were brittle. They could not suggest a function name that fit the context, fill in a complex expression, or understand what the developer was trying to accomplish from incomplete information.

Statistical language models were applied to code as early as the mid-2000s. Naturalness of software, a concept developed by Devanbu and colleagues at UC Davis, proposed that code is repetitive and predictable in ways similar to natural language, and that n-gram models trained on source code could predict the next token with surprising accuracy. These n-gram models served as the theoretical foundation for later neural approaches by establishing that code follows statistical regularities that machine learning can exploit.

The first wave of neural code models applied standard recurrent architectures (LSTMs and GRUs) to source code. Models like code2vec represented code as paths through ASTs and learned embeddings that captured structural properties of functions. These models were useful for specific tasks like clone detection and method name prediction but were not general code generators.

The transformer era changed this. GitHub released Copilot in 2021, based on the Codex model, which demonstrated for the first time that a large autoregressive transformer trained on public code could generate useful, complex functions from natural language descriptions. Codex was trained on roughly 100 billion tokens of code from GitHub and showed that the same architecture and training recipe that worked for natural language scaled naturally to code.

The key insight from Codex was that code generation is fundamentally a language modeling problem. You do not need specialized architectures or code-specific inductive biases in the model. What you need is sufficient scale, high-quality code data, and a tokenizer that handles code well.

The second wave introduced structure-aware objectives and bidirectional understanding. GraphCodeBERT added data flow graphs as additional input structure. CodeBERT used bimodal training with natural language-code pairs. These models showed that incorporating program semantics beyond token sequences improves performance on code understanding tasks.

This generation also introduced the idea that code and natural language are not separate modalities but complementary views of the same underlying intent. A function and its docstring describe the same thing in different languages. Training a model to understand both simultaneously, and to translate between them, produces representations that are more useful than either alone.

The third wave, represented by StarCoder, CodeLlama, and Qwen-Coder, addressed the completion-only limitation of autoregressive models by introducing FIM and expanded the context window to include repository-level information. These models could infill arbitrary code spans and attend to related files, bringing code model capability closer to what a developer experiences when working in a large codebase.

This generation also significantly increased scale: StarCoder was trained on 1 trillion tokens, CodeLlama used 500 billion code-specific tokens for a continued pretraining stage on top of Llama-2, and DeepSeek Coder trained from scratch on 2 trillion tokens. The scale increase, combined with improved data curation, produced models that substantially outperformed the first generation on all benchmarks.

The current state of code models combines large-scale pretraining with instruction tuning and alignment. Models like GPT-4 and Claude are not code-specific, but they achieve strong coding performance because their general capability includes code comprehension and generation as special cases. Code-specific models like DeepSeek Coder V2 and Qwen2.5-Coder achieve competitive performance on coding benchmarks while being more efficient than general-purpose models of similar size.

The trend suggests that the boundary between code models and general language models is dissolving. A model that understands natural language well also understands the natural language in comments, docstrings, and commit messages, which helps it understand code. A model that understands code well can reason about structured data formats like JSON and XML, which helps it handle a wider range of tasks.

Beyond the left-to-right language modeling objective and FIM, several additional pre-training objectives have been explored for code models. Comparing these objectives helps explain why different code models have different strengths and weaknesses.

The base objective for all autoregressive code models is the same cross-entropy loss used in standard language model pretraining. For a training sequence of tokens $t_1, t_2, \ldots, t_n$, the model maximizes:

where:

• $t_i$ is the $i$-th token in the sequence
• $P(t_i \mid t_1, \ldots, t_{i-1}; \theta)$ is the model's predicted probability for token $t_i$ given all preceding tokens
• $\theta$ represents the model parameters
• $n$ is the total number of tokens in the sequence

This objective applies identically to code and natural language. What changes is the data distribution: training on code data causes the model to allocate capacity toward code-relevant patterns. The same transformer architecture that learns to predict the next word in an English sentence learns to predict the next token in a Python function.

One subtle difference from natural language training is that code documents are typically much shorter than web pages but have much higher information density per token. A 50-line Python function may contain more unique structural patterns than a 500-word news article. This affects how long training sequences should be and how documents should be packed into fixed-length training batches. Most code models train with longer context windows than natural language models of similar size, because code understanding often requires attending across hundreds of lines.

The CLM objective has one important property for code: it trains the model to predict every token, including syntactically redundant ones like closing parentheses and brackets. This might seem wasteful, since a deterministic parser could generate these, but it helps the model learn the syntactic structure of programs. Predicting the matching bracket to a def ( requires understanding scope and nesting, which is also the structure the model needs to generate semantically coherent code.

BERT-style masked language modeling has been applied to code in models like CodeBERT, GraphCodeBERT, and UniXcoder. The objective masks a random subset of tokens and requires the model to predict them from bidirectional context:

where:

• $\mathcal{M}$ is the set of masked token positions
• $t_{\setminus \mathcal{M}}$ denotes the unmasked tokens, giving full bidirectional context
• $P(t_i \mid t_{\setminus \mathcal{M}}; \theta)$ is the model's probability for the masked token given all unmasked context

MLM-based code models excel at tasks that benefit from bidirectional context: code search, clone detection, vulnerability detection, and code classification. They are not directly usable for generation but serve as powerful encoders. When you need to embed a code snippet for search or clustering, a bidirectional encoder trained with MLM typically outperforms a causal decoder for the same number of parameters.

The tradeoff with MLM is architectural: a model trained with bidirectional attention cannot be directly used for autoregressive generation. This is why the field has largely moved toward causal models for code generation, with MLM-based encoders occupying a separate niche for understanding and retrieval tasks.

Beyond token-level prediction, some code models have explored structure-aware objectives that reflect the distinctive properties of programs:

Identifier masking: Instead of masking random tokens, mask only identifier names. This forces the model to predict what a variable or function is called given its usage context, which requires understanding program semantics rather than just surface syntax. A model that can predict the correct name for an identifier has learned something about the role of that identifier in the program's logic.

Docstring-to-code generation: Given a function's docstring, predict its implementation. This is essentially few-shot instruction following built into the pretraining objective, and it directly trains the model for code generation from natural language specifications. Models trained on this objective perform better on benchmark tasks like HumanEval that require generating a function from its description.

Code-to-docstring generation: Given a function implementation, generate its docstring. This is the reverse of docstring-to-code and trains the model to produce natural language explanations of code behavior. Models trained on this objective tend to be better at code summarization and explanation tasks.

Data flow prediction: Predict the data flow graph structure, which variables influence which others, in addition to token sequences. GraphCodeBERT added data flow as an auxiliary training signal, encoding which variables are computed from which other variables, and showed significant improvements on code search and code translation. The intuition is that data flow represents the semantic essence of a function's behavior, independent of its syntactic representation.

Type prediction: Predict the types of variables and function arguments in dynamically typed languages like Python. This encourages the model to reason about data flow and function contracts. A model that can infer that a variable should be an np.ndarray based on how it is used has learned something about the implicit type system that makes Python programs correct.

Modern code models frequently combine multiple objectives. The intuition is that different objectives develop different capabilities: CLM trains generation, MLM trains bidirectional understanding, and structural objectives develop semantic understanding. Combining them produces models with complementary strengths that no single objective would achieve alone.

However, combining objectives requires careful balancing. If the structural objectives dominate, the model may sacrifice generation quality. If CLM dominates, the structural understanding remains shallow. The loss weighting, batch composition, and gradient interference between objectives all influence the final model's behavior.

One common approach is alternating objective sampling: each training batch is drawn from a single objective, and the objective is sampled randomly according to a fixed schedule. This avoids the gradient conflict that arises when objectives with incompatible gradient directions are summed in the same backward pass. It also makes hyperparameter tuning simpler, since each objective's learning rate and batch size can be set independently.

CodeT5, developed by Salesforce, combined bimodal generation (code-to-text and text-to-code), identifier masking, and masked span prediction in a unified encoder-decoder architecture. The multi-task training yielded models that outperformed single-objective baselines on both generation and understanding benchmarks. The encoder-decoder architecture is well-suited to multi-objective training because the encoder can be shared across understanding tasks while the decoder is used for generation.

UniXcoder used a combination of MLM, code completion (CLM), and cross-modal contrastive learning between code and comments to produce representations useful for both generation and retrieval tasks. The contrastive objective pulled together representations of a function and its natural language summary in a shared embedding space, enabling the model to be used directly for code search by embedding a natural language query and finding the nearest code embedding.

One objective that has received growing attention is training on execution outcomes. Rather than treating code as a sequence of tokens to predict, execution-based training uses the results of running code as a training signal.

The idea is straightforward: a model that generates code should prefer code that runs correctly over code that does not. If you have a dataset of programming problems with unit tests, you can run generated code against the tests and use the pass/fail outcomes to provide feedback. Models like AlphaCode and CodeRL incorporated execution feedback through reinforcement learning from code execution, treating test passage as a reward signal.

Execution-based training grounds the model's behavior in what code does, rather than how it looks. A model trained purely on token prediction might generate code that looks syntactically correct but fails basic sanity checks. Execution feedback corrects this by penalizing code that fails at runtime, even when it appears superficially reasonable.

The practical challenge is computational cost: running code is orders of magnitude more expensive than computing a token prediction loss. Each generated candidate must be sandboxed, executed, and its output compared to expected results. At scale, this requires large execution clusters and careful resource management to avoid both slowness and security issues from running untrusted generated code.

One underappreciated aspect of code model training is curriculum design: the order in which different kinds of code are presented during training. Naive approaches shuffle all training examples uniformly, but structured curricula can improve both training efficiency and final model quality.

The intuition behind curriculum learning is that models learn more efficiently when examples are presented in order of increasing difficulty. A model that has learned simple function definitions and control flow will benefit more from seeing complex class hierarchies and decorators than a model that has never seen basic Python. Presenting all difficulty levels uniformly may cause the model to waste capacity on examples it cannot yet learn from.

In practice, most large-scale code model training does not use explicit curriculum learning, because sorting billions of files by difficulty is itself a non-trivial problem. Instead, some teams use implicit curricula: placing documentation-rich, well-tested, and commonly starred repositories earlier in training, on the assumption that popular, well-maintained repositories represent higher-quality code. The continued pretraining stage of CodeLlama, which used a 2 trillion token mix of code-heavy data, applied a version of this by weighting high-quality coding datasets more heavily in the final training stage than in the base pretraining stage.

Context length curriculum is more commonly used: models are trained with shorter context windows early in training and longer windows later. This is partly a hardware efficiency optimization (shorter sequences fit more examples in memory) and partly a curriculum effect (shorter-context examples are simpler). Extending the context window during training requires scaling the positional encoding, which is why techniques like YaRN and position interpolation described in earlier chapters are directly relevant to code model training.

A practical detail with significant effects on code model quality is how training documents are packed into fixed-length input sequences. At training time, examples from the dataset are concatenated into a fixed-length token sequence (typically 2,048 or 4,096 tokens) before being fed to the model. Naive concatenation can cause the model to attend across document boundaries: the beginning of one file becomes visible to the attention mechanism of the previous file's last token, creating spurious dependencies.

The standard fix is to insert end-of-document tokens between concatenated documents, typically a special <EOD> or <|endoftext|> token, and to use an attention mask that prevents attention from crossing document boundaries. When this boundary masking is applied, the model never sees tokens from the previous document when processing a given document, preserving the independence assumption.

For code models, proper boundary handling is especially important because code structure is tightly coupled. A Python file that ends with a def statement should not have the model attend to the next file's variable definitions as if they were part of the same module. Without boundary masking, the model might learn spurious patterns from accidental co-occurrences of code from unrelated files that happen to appear adjacent in the packed sequence.

Let's build a simplified data pipeline that demonstrates the key steps: loading code files, applying filters, and constructing a FIM training objective. We'll use a small in-memory dataset to keep execution fast, but the patterns generalize directly to large-scale pipelines.

After filtering and deduplication, our corpus is smaller but higher quality. In production, these steps typically reduce a raw GitHub scrape by 70 to 90 percent, while the remaining data is cleaner and more representative of real, idiomatic code.

Now let's implement the fill-in-the-middle transformation. We'll apply it to a realistic code snippet and show the resulting training format for both the PSM and SPM variants.

The training batch is now a mix of FIM and standard causal LM examples. The model learns to handle both: standard left-to-right completion from prefix-only contexts, and infilling when prefix and suffix are both present.

Let's examine how tokenization behavior differs for code versus natural language, using the tokenizers library to implement a simple BPE tokenizer trained on our code corpus.

Code LLM training faces several distinctive challenges that do not have clean solutions. Understanding these limitations is important for both building code models and for calibrating what to expect from them in production.

The most fundamental limitation is the signal-to-noise ratio of public code. The vast majority of GitHub repositories are personal projects, student assignments, or abandoned experiments that were never intended to serve as exemplary code. Even after aggressive filtering, a model trained on public code may develop subtle biases: it sees more Python web frameworks than embedded systems code, more JavaScript front-end patterns than systems programming, and more English-language identifiers and comments than code from non-English-speaking developer communities.

Filtering heuristics like line length and alphabetic ratio help with obvious garbage but cannot detect subtler quality issues: incorrect algorithms presented as correct implementations, security vulnerabilities that appear syntactically valid, deprecated API patterns that were once idiomatic but no longer represent best practice, or code that handles happy paths correctly but fails silently on edge cases. This means that code LLMs can confidently generate code that is syntactically plausible but semantically wrong or insecure. A function generated from a description of a binary search might work for most inputs but have an off-by-one error in the edge case, because the training data included similar off-by-one errors.

The field has responded with execution-based verification: filtering training data by running code and checking that it produces expected outputs. CodeContests and MBPP-style datasets use this approach for benchmark construction, but execution-based filtering at the scale of billions of code files remains computationally prohibitive. The cost of sandboxing and running billions of code files is orders of magnitude higher than the cost of text-based heuristics.

FIM is a powerful technique, but it does not give the model true bidirectional attention. At inference time, a FIM model uses its suffix context to condition generation, but the attention is still causal: the suffix tokens cannot attend to the infill tokens being generated. This means the model must "anticipate" what it will write in order to connect correctly with the suffix, rather than observing the infill and the suffix simultaneously as a bidirectional model would.

This contrasts with masked language models like BERT, which use bidirectional attention. When BERT predicts a masked token, all surrounding tokens (both before and after) directly attend to the position being predicted. A FIM model has read the suffix before generating the infill, but each generated infill token can only attend to previously generated infill tokens plus the full prefix-suffix context, not to future infill tokens.

The practical consequence is that FIM models sometimes produce infills that are locally plausible but fail to terminate or transition correctly into the suffix. IDE completion systems mitigate this by re-ranking generated completions using a separate model that scores their coherence with the surrounding context, or by generating multiple candidates and selecting the one that most naturally connects to the suffix.

Code models are most commonly evaluated using HumanEval (pass@k on 164 Python programming problems) and similar benchmarks. These benchmarks have been widely criticized for being too narrow: they test algorithmic problem solving in Python but say little about how well a model handles multi-file TypeScript projects, debugging sessions, or infrastructure-as-code tasks that represent the majority of real software development work.

HumanEval problems are short, self-contained, and algorithmically oriented, the kind of task that appears on LeetCode. Real-world code assistance is more often about understanding a 2,000-line file, tracing why a specific function returns the wrong value, or generating a configuration file that correctly integrates with an existing system. These tasks require contextual reasoning that HumanEval does not assess.

Models optimized for HumanEval performance may be overfitted to its specific problem style rather than being more useful code assistants. The field is actively developing richer evaluation suites: SWE-bench requires resolving real GitHub issues, CrossCodeEval tests multi-language and multi-file evaluation, and BigCodeBench covers more diverse task types including data processing, scientific computing, and system administration. These benchmarks are harder to game and better reflect the distribution of tasks where developers need help.

Public code repositories contain private information: API keys accidentally committed, passwords in configuration files, personal email addresses in copyright headers, and proprietary algorithms published without understanding they would become training data. Code models can and do memorize these patterns and reproduce them when prompted with similar context.

This has prompted both technical and policy responses. Technically, differential privacy training noise, membership inference testing, and post-training filtering for known sensitive patterns have been explored. The challenge with differential privacy is that the privacy budget required to protect individual examples typically degrades model quality significantly. Post-training filtering can catch specific known secrets (like public API key formats) but cannot detect novel sensitive content.

At the policy level, organizations including GitHub, Google, and BigCode have developed opt-out registries and license-aware data curation practices. The Stack allows developers to request removal of their repositories from the training data. The tension between model capability (more data is almost always better) and responsible training data use remains an active area of debate. There is no technical solution that perfectly balances these concerns, and the field continues to develop better approaches.

Even with upsampling, low-resource programming languages remain difficult to model well. A language like Elm or Zig, which has perhaps 50,000 public repositories versus Python's 10 million, simply has insufficient training data to learn all the idioms and patterns that make Elm or Zig code good. The model learns enough to produce syntactically valid code in these languages but lacks the depth of understanding it has for high-resource languages.

This creates a troubling asymmetry: developers working in niche languages often need the most help (because community resources and Stack Overflow answers are scarcer) but receive the least capable assistance. Some work has explored cross-lingual transfer: teaching a model Rust using its understanding of C (which is more common) and the formal semantics they share. But this remains an open research challenge.

Training a code LLM requires careful attention at every stage of the pipeline, from data collection through tokenization design to training objectives.

On the data side, effective code training starts with high-quality sources (repositories, documentation, notebooks) and applies aggressive filtering for quality and deduplication. Near-deduplication with MinHash removes the flood of identical copies that inflate raw corpus sizes without adding signal. Language balancing ensures the model develops competence across programming languages, not just Python and JavaScript. Repository-level context organization, while technically complex, provides the cross-file semantic relationships that make a model useful for real-world coding rather than toy examples.

Code tokenization benefits from BPE vocabularies trained on code-specific corpora, which naturally capture identifier structure, language keywords, and operator sequences as single tokens. Byte-level BPE ensures no source file is ever out-of-vocabulary. Vocabulary sizes in the range of 32,000 to 100,000 tokens balance sequence length, embedding table size, and expressiveness. The choice of tokenizer directly affects how much context the model can attend to, since shorter token sequences allow more code to fit within the context window.

Fill-in-the-middle training transforms what completion models can do by teaching them to use both prefix and suffix context. The PSM and SPM variants rearrange document order during training using special sentinel tokens, allowing the model to infill any span rather than only complete text forward from a prefix. FIM works because it reduces an infilling task to a standard autoregressive prediction task through document reordering, requiring no architectural changes. Training with approximately 50 percent FIM examples preserves forward-completion quality while adding infilling capability.

Pre-training objectives range from standard causal language modeling to masked language modeling (for encoder models), identifier masking, bimodal code-text generation, and execution-based feedback. Multi-objective training on objectives like CodeT5's combination of span masking and bimodal generation can improve performance on both generation and understanding tasks. Execution feedback provides the strongest quality signal but at significant computational cost.

The practical limitations are real: data quality at scale remains unsolved, FIM is not true bidirectionality, benchmarks like HumanEval do not fully capture real-world code assistance quality, memorization of sensitive information from public repositories poses a concrete risk, and low-resource programming languages remain poorly served. Understanding these limitations helps calibrate what code LLMs can reliably do and where human oversight remains essential.

These design decisions are tightly connected. The choice of tokenizer affects how much context the model can see for a given memory budget, which affects how well FIM works for long files. The quality of training data affects how well multi-objective training generalizes, because diverse objectives only help if the data is diverse enough to provide distinct learning signal for each. Repository-level context only improves results if the tokenizer is efficient enough to leave room for multiple files in the context window. Good code model training requires optimizing all these components together, not independently.

The next chapter builds on this foundation by examining how instruction tuning and alignment training transform a pretrained code model into an assistant that follows developer intent, responds to natural language task descriptions, and avoids generating insecure or harmful code.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about code LLM training.