Code LLM Training › Code Pre-Training Objectives

Beyond the left-to-right language modeling objective and FIM, several additional pre-training objectives have been explored for code models. Comparing these objectives helps explain why different code models have different strengths and weaknesses.

The base objective for all autoregressive code models is the same cross-entropy loss used in standard language model pretraining. For a training sequence of tokens $t_1, t_2, \ldots, t_n$, the model maximizes:

$$\mathcal{L}_{\text{CLM}} = -\frac{1}{n} \sum_{i=1}^{n} \log P(t_i \mid t_1, t_2, \ldots, t_{i-1}; \theta)$$

where:

• $t_i$ is the $i$-th token in the sequence
• $P(t_i \mid t_1, \ldots, t_{i-1}; \theta)$ is the model's predicted probability for token $t_i$ given all preceding tokens
• $\theta$ represents the model parameters
• $n$ is the total number of tokens in the sequence

This objective applies identically to code and natural language. What changes is the data distribution: training on code data causes the model to allocate capacity toward code-relevant patterns. The same transformer architecture that learns to predict the next word in an English sentence learns to predict the next token in a Python function.

One subtle difference from natural language training is that code documents are typically much shorter than web pages but have much higher information density per token. A 50-line Python function may contain more unique structural patterns than a 500-word news article. This affects how long training sequences should be and how documents should be packed into fixed-length training batches. Most code models train with longer context windows than natural language models of similar size, because code understanding often requires attending across hundreds of lines.

The CLM objective has one important property for code: it trains the model to predict every token, including syntactically redundant ones like closing parentheses and brackets. This might seem wasteful, since a deterministic parser could generate these, but it helps the model learn the syntactic structure of programs. Predicting the matching bracket to a def ( requires understanding scope and nesting, which is also the structure the model needs to generate semantically coherent code.

BERT-style masked language modeling has been applied to code in models like CodeBERT, GraphCodeBERT, and UniXcoder. The objective masks a random subset of tokens and requires the model to predict them from bidirectional context:

$$\mathcal{L}_{\text{MLM}} = -\sum_{i \in \mathcal{M}} \log P(t_i \mid t_{\setminus \mathcal{M}}; \theta)$$

where:

• $\mathcal{M}$ is the set of masked token positions
• $t_{\setminus \mathcal{M}}$ denotes the unmasked tokens, giving full bidirectional context
• $P(t_i \mid t_{\setminus \mathcal{M}}; \theta)$ is the model's probability for the masked token given all unmasked context

MLM-based code models excel at tasks that benefit from bidirectional context: code search, clone detection, vulnerability detection, and code classification. They are not directly usable for generation but serve as powerful encoders. When you need to embed a code snippet for search or clustering, a bidirectional encoder trained with MLM typically outperforms a causal decoder for the same number of parameters.

The tradeoff with MLM is architectural: a model trained with bidirectional attention cannot be directly used for autoregressive generation. This is why the field has largely moved toward causal models for code generation, with MLM-based encoders occupying a separate niche for understanding and retrieval tasks.

Beyond token-level prediction, some code models have explored structure-aware objectives that reflect the distinctive properties of programs:

Identifier masking: Instead of masking random tokens, mask only identifier names. This forces the model to predict what a variable or function is called given its usage context, which requires understanding program semantics rather than just surface syntax. A model that can predict the correct name for an identifier has learned something about the role of that identifier in the program's logic.

Docstring-to-code generation: Given a function's docstring, predict its implementation. This is essentially few-shot instruction following built into the pretraining objective, and it directly trains the model for code generation from natural language specifications. Models trained on this objective perform better on benchmark tasks like HumanEval that require generating a function from its description.

Code-to-docstring generation: Given a function implementation, generate its docstring. This is the reverse of docstring-to-code and trains the model to produce natural language explanations of code behavior. Models trained on this objective tend to be better at code summarization and explanation tasks.

Data flow prediction: Predict the data flow graph structure, which variables influence which others, in addition to token sequences. GraphCodeBERT added data flow as an auxiliary training signal, encoding which variables are computed from which other variables, and showed significant improvements on code search and code translation. The intuition is that data flow represents the semantic essence of a function's behavior, independent of its syntactic representation.

Type prediction: Predict the types of variables and function arguments in dynamically typed languages like Python. This encourages the model to reason about data flow and function contracts. A model that can infer that a variable should be an np.ndarray based on how it is used has learned something about the implicit type system that makes Python programs correct.

Modern code models frequently combine multiple objectives. The intuition is that different objectives develop different capabilities: CLM trains generation, MLM trains bidirectional understanding, and structural objectives develop semantic understanding. Combining them produces models with complementary strengths that no single objective would achieve alone.

However, combining objectives requires careful balancing. If the structural objectives dominate, the model may sacrifice generation quality. If CLM dominates, the structural understanding remains shallow. The loss weighting, batch composition, and gradient interference between objectives all influence the final model's behavior.

One common approach is alternating objective sampling: each training batch is drawn from a single objective, and the objective is sampled randomly according to a fixed schedule. This avoids the gradient conflict that arises when objectives with incompatible gradient directions are summed in the same backward pass. It also makes hyperparameter tuning simpler, since each objective's learning rate and batch size can be set independently.

CodeT5, developed by Salesforce, combined bimodal generation (code-to-text and text-to-code), identifier masking, and masked span prediction in a unified encoder-decoder architecture. The multi-task training yielded models that outperformed single-objective baselines on both generation and understanding benchmarks. The encoder-decoder architecture is well-suited to multi-objective training because the encoder can be shared across understanding tasks while the decoder is used for generation.

UniXcoder used a combination of MLM, code completion (CLM), and cross-modal contrastive learning between code and comments to produce representations useful for both generation and retrieval tasks. The contrastive objective pulled together representations of a function and its natural language summary in a shared embedding space, enabling the model to be used directly for code search by embedding a natural language query and finding the nearest code embedding.

One objective that has received growing attention is training on execution outcomes. Rather than treating code as a sequence of tokens to predict, execution-based training uses the results of running code as a training signal.

The idea is straightforward: a model that generates code should prefer code that runs correctly over code that does not. If you have a dataset of programming problems with unit tests, you can run generated code against the tests and use the pass/fail outcomes to provide feedback. Models like AlphaCode and CodeRL incorporated execution feedback through reinforcement learning from code execution, treating test passage as a reward signal.

Execution-based training grounds the model's behavior in what code does, rather than how it looks. A model trained purely on token prediction might generate code that looks syntactically correct but fails basic sanity checks. Execution feedback corrects this by penalizing code that fails at runtime, even when it appears superficially reasonable.

The practical challenge is computational cost: running code is orders of magnitude more expensive than computing a token prediction loss. Each generated candidate must be sandboxed, executed, and its output compared to expected results. At scale, this requires large execution clusters and careful resource management to avoid both slowness and security issues from running untrusted generated code.

One underappreciated aspect of code model training is curriculum design: the order in which different kinds of code are presented during training. Naive approaches shuffle all training examples uniformly, but structured curricula can improve both training efficiency and final model quality.

The intuition behind curriculum learning is that models learn more efficiently when examples are presented in order of increasing difficulty. A model that has learned simple function definitions and control flow will benefit more from seeing complex class hierarchies and decorators than a model that has never seen basic Python. Presenting all difficulty levels uniformly may cause the model to waste capacity on examples it cannot yet learn from.

In practice, most large-scale code model training does not use explicit curriculum learning, because sorting billions of files by difficulty is itself a non-trivial problem. Instead, some teams use implicit curricula: placing documentation-rich, well-tested, and commonly starred repositories earlier in training, on the assumption that popular, well-maintained repositories represent higher-quality code. The continued pretraining stage of CodeLlama, which used a 2 trillion token mix of code-heavy data, applied a version of this by weighting high-quality coding datasets more heavily in the final training stage than in the base pretraining stage.

Context length curriculum is more commonly used: models are trained with shorter context windows early in training and longer windows later. This is partly a hardware efficiency optimization (shorter sequences fit more examples in memory) and partly a curriculum effect (shorter-context examples are simpler). Extending the context window during training requires scaling the positional encoding, which is why techniques like YaRN and position interpolation described in earlier chapters are directly relevant to code model training.

A practical detail with significant effects on code model quality is how training documents are packed into fixed-length input sequences. At training time, examples from the dataset are concatenated into a fixed-length token sequence (typically 2,048 or 4,096 tokens) before being fed to the model. Naive concatenation can cause the model to attend across document boundaries: the beginning of one file becomes visible to the attention mechanism of the previous file's last token, creating spurious dependencies.

The standard fix is to insert end-of-document tokens between concatenated documents, typically a special <EOD> or <|endoftext|> token, and to use an attention mask that prevents attention from crossing document boundaries. When this boundary masking is applied, the model never sees tokens from the previous document when processing a given document, preserving the independence assumption.

For code models, proper boundary handling is especially important because code structure is tightly coupled. A Python file that ends with a def statement should not have the model attend to the next file's variable definitions as if they were part of the same module. Without boundary masking, the model might learn spurious patterns from accidental co-occurrences of code from unrelated files that happen to appear adjacent in the packed sequence.