Code LLM Training › Summary

Training a code LLM requires careful attention at every stage of the pipeline, from data collection through tokenization design to training objectives.

On the data side, effective code training starts with high-quality sources (repositories, documentation, notebooks) and applies aggressive filtering for quality and deduplication. Near-deduplication with MinHash removes the flood of identical copies that inflate raw corpus sizes without adding signal. Language balancing ensures the model develops competence across programming languages, not just Python and JavaScript. Repository-level context organization, while technically complex, provides the cross-file semantic relationships that make a model useful for real-world coding rather than toy examples.

Code tokenization benefits from BPE vocabularies trained on code-specific corpora, which naturally capture identifier structure, language keywords, and operator sequences as single tokens. Byte-level BPE ensures no source file is ever out-of-vocabulary. Vocabulary sizes in the range of 32,000 to 100,000 tokens balance sequence length, embedding table size, and expressiveness. The choice of tokenizer directly affects how much context the model can attend to, since shorter token sequences allow more code to fit within the context window.

Fill-in-the-middle training transforms what completion models can do by teaching them to use both prefix and suffix context. The PSM and SPM variants rearrange document order during training using special sentinel tokens, allowing the model to infill any span rather than only complete text forward from a prefix. FIM works because it reduces an infilling task to a standard autoregressive prediction task through document reordering, requiring no architectural changes. Training with approximately 50 percent FIM examples preserves forward-completion quality while adding infilling capability.

Pre-training objectives range from standard causal language modeling to masked language modeling (for encoder models), identifier masking, bimodal code-text generation, and execution-based feedback. Multi-objective training on objectives like CodeT5's combination of span masking and bimodal generation can improve performance on both generation and understanding tasks. Execution feedback provides the strongest quality signal but at significant computational cost.

The practical limitations are real: data quality at scale remains unsolved, FIM is not true bidirectionality, benchmarks like HumanEval do not fully capture real-world code assistance quality, memorization of sensitive information from public repositories poses a concrete risk, and low-resource programming languages remain poorly served. Understanding these limitations helps calibrate what code LLMs can reliably do and where human oversight remains essential.

These design decisions are tightly connected. The choice of tokenizer affects how much context the model can see for a given memory budget, which affects how well FIM works for long files. The quality of training data affects how well multi-objective training generalizes, because diverse objectives only help if the data is diverse enough to provide distinct learning signal for each. Repository-level context only improves results if the tokenizer is efficient enough to leave room for multiple files in the context window. Good code model training requires optimizing all these components together, not independently.

The next chapter builds on this foundation by examining how instruction tuning and alignment training transform a pretrained code model into an assistant that follows developer intent, responds to natural language task descriptions, and avoids generating insecure or harmful code.