Code LLM Training › Code Training Data

The quality and composition of training data is the single most consequential decision in building a code model. Unlike natural language pretraining, where any sufficiently large corpus of web text provides useful signal, code data requires deliberate filtering, deduplication, and structural understanding. The difference matters because code has properties that web text does not: it can be syntactically correct yet semantically wrong, it exists in enormous quantities of low-quality form (generated, copied, or trivially repetitive), and its quality can sometimes be verified automatically by running it.

The primary source of code training data is public version control hosting. GitHub, GitLab, and similar platforms host hundreds of millions of repositories containing code in thousands of programming languages. The GitHub Code dataset, used in early Codex training, contained roughly 54 million public repositories. The Stack, assembled by BigCode, covers over 350 programming languages and reached approximately 3 terabytes of source code after initial filtering.

Beyond repositories, code data comes from several complementary sources, each offering a different kind of learning signal:

• Documentation and tutorials: Official language documentation, framework guides, and tutorials contain correct, idiomatic code accompanied by natural language explanations. This is especially valuable because it pairs code with intent, which is precisely the kind of alignment a code assistant needs. A model trained on documentation learns how functions are implemented and what they are for.
• Competitive programming: Platforms like Codeforces, LeetCode, and HackerRank provide algorithmically challenging problems with multiple correct solutions, often accompanied by comments explaining the approach. This data pushes the model to learn algorithmic reasoning rather than only rote code patterns.
• Notebooks: Jupyter notebooks from GitHub and Kaggle provide code interleaved with markdown explanations, giving fine-grained alignment between prose and code at the cell level. Models trained on notebook data tend to produce better-explained code and are more capable of describing what a block of code does.
• Commit histories: Diffs between commits capture how code evolves, which can be used to train models on code editing and bug fixing. A commit that changes a function from an incorrect to a correct version is a golden example of the editing task that real developers encounter.
• Issues and pull requests: Discussion threads on GitHub pair natural language problem descriptions with code patches, enabling instruction-following for code tasks. These are especially valuable for fine-tuning, since they closely resemble the conversational format in which developers interact with code assistants.

The ratio between these sources matters. Models trained primarily on raw source files learn syntax and structure but may struggle with natural language descriptions of problems. Models with more documentation and notebook data tend to generalize better to instruction-following and code explanation tasks. The early Codex model was criticized for being excellent at generating plausible code but inconsistent at following docstring-based specifications, a limitation attributed in part to the dominance of raw source files in its training mix.

Programming languages vary enormously in how much public code exists. Python, JavaScript, Java, and C/C++ dominate public repositories by several orders of magnitude. Languages like Rust, Haskell, or niche domain-specific languages have far less public code available. Languages like Fortran, COBOL, or assembly exist in even smaller quantities.

Naive training on the raw distribution would produce a model that is excellent at Python and JavaScript but nearly useless for Rust, Go, or TypeScript. This is a serious problem for tools marketed as general-purpose code assistants, since developers using those languages need help just as much as Python developers do.

Two strategies address this imbalance:

Upsampling rare languages: Multiply the effective frequency of underrepresented languages during training. If Rust represents 0.1% of the raw corpus but you want similar performance to Python, you sample Rust files at 10 to 50 times their natural frequency. The tradeoff is that upsampling can cause overfitting on the smaller Rust corpus, since the model sees the same files repeatedly. A Rust file might be presented hundreds of times across training while a Python file appears only once.

Minimum language thresholds: Set a minimum token count per language below which the language is excluded entirely. This prevents the model from learning partial, low-quality representations of languages with only a few hundred files, which are worse than learning nothing at all. A few hundred Fortran files will teach the model some Fortran syntax but may also introduce subtle errors or non-idiomatic patterns that are hard to detect.

The Stack v2 and similar recent datasets take a principled approach, targeting rough parity between languages with substantial public code while excluding languages with fewer than a few thousand repositories. Research from BigCode found that language balance significantly affects downstream benchmark performance on multilingual code evaluation. Interestingly, cross-language transfer also matters: a model with strong Python understanding can generalize better to TypeScript or Kotlin than a model trained on fewer total languages, even if the latter was trained on more examples of each specific language. This transfer effect suggests that learning the common structure of programs across languages provides a useful inductive bias.

Raw repository data contains enormous amounts of noise. A repository scraped from GitHub might include auto-generated code from build systems or protobuf compilers, minified JavaScript where all whitespace and identifiers have been stripped, binary files with .py or .js extensions, duplicate code copied between repositories with minor modifications, low-quality educational files with repetitive trivial content, and files that are primarily comments with almost no actual implementation.

Each of these adds to training time and data storage while providing minimal learning signal. More dangerously, some kinds of noise actively degrade model quality. Minified code teaches the model that long single-line statements without whitespace are valid Python. Auto-generated boilerplate teaches the model that repetitive patterns with no semantic meaning are normal. These biases show up as quirks in generated code that are difficult to remove after the fact.

Standard filtering steps include several complementary approaches:

Exact deduplication: Remove identical files across repositories. Surprisingly, a large fraction of public code is direct copies. The Stack found that approximately 70% of files could be removed through near-deduplication, leaving only the highest-quality representative copy. The sheer scale of this duplication reflects how common it is for developers to copy-paste starter templates, configuration files, or example implementations.

Near-deduplication using MinHash: Exact deduplication misses slightly modified copies. MinHash locality-sensitive hashing identifies files with above a certain Jaccard similarity threshold, typically 0.8 or higher, and keeps only one representative. This dramatically reduces memorization of boilerplate. Without near-deduplication, a model may memorize common license headers, README templates, or configuration snippets at the expense of learning more diverse code patterns.

Quality heuristics: Filter files by metrics including line length distribution (very long average lines indicate minified code), alphabetic character fraction (very low fraction indicates binary or generated content), number of unique lines as a fraction of total lines (very low fraction indicates repetitive files), and comment-to-code ratio (files with no code lines are documentation, not implementation). These heuristics are fast to compute at scale and remove the most obvious garbage.

License filtering: Many models filter to permissively licensed code such as MIT, Apache 2.0, and BSD to reduce legal risk. The Stack provides license metadata for this purpose. This filtering removes a meaningful fraction of highly useful code, since many projects use restrictive licenses, but it is considered essential for commercial deployment.

One subtle but important aspect of code data is that files rarely exist in isolation. A Python function that calls np.array() is meaningless without knowing that numpy is imported. A TypeScript file uses types defined in adjacent files. A Rust function requires crate dependencies. Understanding code at a file level misses the semantic relationships required for programs to work.

This context problem has direct consequences for how training data is organized. If a model never sees that compute_loss is called with a specific signature in train.py and defined with a matching signature in model.py, it cannot learn the conventions that govern cross-file consistency. It may learn to write plausible function definitions, but it will not learn to write functions that fit naturally into a codebase. The same observation applies to type information: knowing that a function expects an np.ndarray rather than a Python list is essential for generating code that composes correctly, but this information often lives in type stubs or adjacent files.

Early code models like Codex trained on individual files without repository context, which limited their ability to use multi-file context. Later models like GitHub Copilot and StarCoder introduced repository-level context, including adjacent file contents, import graphs, and function signatures from related modules. The AlphaCode dataset organized training data at the problem level, making sure the model saw complete problem specifications alongside solution code.

Repository-level data organization is technically more challenging than file-level organization. It requires crawling the repository's dependency graph, resolving import paths, and deciding how to pack related files together within the context window. Context packing choices matter: including too many irrelevant files dilutes the signal, while including too few misses important cross-file relationships. But the performance gains on real-world coding tasks are substantial enough that repository-level context has become standard practice in production code LLMs.

To make filtering concrete, consider a raw Python file scraped from GitHub. It might be 800 lines long with a one-line copyright header, 700 lines of auto-generated protobuf binding code that looks like class ResponseType_pb2: __slots__ = ['field_a', 'field_b'...] repeated with minor variations, 80 lines of actual logic, and 20 blank lines. The unique-line ratio is approximately 0.12 (most lines are near-copies), so a threshold of 0.4 would reject it. Running it through the alphabetic ratio filter would also likely reject it because of the abundance of brackets and underscores in the generated code.

After filtering, you retain the 80 lines of actual logic, but only if those 80 lines appear in a different file that is not buried inside the auto-generated file. This example illustrates why filtering is effective: it removes the flood of repetitive content while keeping files where most lines are semantically distinct.