Code LLM Training › Code Tokenization

Tokenization for code differs from natural language tokenization in ways that go beyond just applying BPE to source text. The structure of identifiers, whitespace sensitivity in indented languages, and the prevalence of long token sequences all affect how tokenization choices translate into model capability. A poorly designed tokenizer does not just produce longer sequences, it actively destroys the semantic structure of code in ways that make learning harder.

Consider the identifier calculateMeanSquaredError. In natural language, this would almost never appear, but splitting it at byte boundaries or random subword boundaries is semantically destructive. The identifier has meaningful internal structure: calculate, Mean, Squared, Error. A tokenizer that splits it as cal|culat|eMean|Squared|Error loses the semantic decomposition that would help a model understand what the function does. Worse, the same identifier split differently at different occurrences in the training data produces inconsistent representations, making the model's job harder.

CamelCase and snake_case identifiers have consistent internal structure that tokenizers should ideally respect. Similarly, compound operators like ->, ::, =>, and ** have unified semantic meanings that should be preserved as single tokens. A model that sees -> split as - and > has a harder time learning that it is a member access operator.

Three properties distinguish effective code tokenizers:

Identifier awareness: The tokenizer should be trained on code data so its vocabulary naturally includes common identifiers, language keywords, and operator sequences as single tokens. A vocabulary trained on web text will not include np.array or torch.nn.Module, while a code-tuned vocabulary may encode these as one or two tokens, dramatically reducing sequence lengths for common patterns.

Whitespace handling: In Python and other indentation-sensitive languages, leading spaces carry syntactic meaning. A tokenizer must represent indentation faithfully. Many code tokenizers reserve special tokens for common indentation patterns such as two spaces, four spaces, eight spaces, and a tab to avoid exploding sequence length in deeply nested code. Without this, a function nested four levels deep might spend a substantial fraction of its token budget just on leading spaces.

Number representation: Source code contains many numeric literals with specific formats. 0x1F3A (hexadecimal), 3.14159 (float), 1e-5 (scientific notation), and 0b1010 (binary) are all valid number literals in most languages. A tokenizer should handle these consistently rather than producing radically different tokenizations for the same numeric value in different notations.

Larger vocabularies reduce sequence length and allow more context to fit within the model's context window for a given amount of source code, but they increase the size of the embedding table and output projection matrix. For a model with hidden dimension $d$ and vocabulary size $V$, the output projection has $V \times d$ parameters. Doubling the vocabulary doubles this component's parameter count. For a model with $d = 4096$ and $V = 100{,}000$, the output projection alone has 409 million parameters, which is larger than some full models.

Small vocabularies (like character-level tokenization) produce very long sequences that are expensive to process, since attention cost scales quadratically with sequence length, but they can represent any code without out-of-vocabulary tokens.

Most production code models use BPE vocabularies in the range of 32,000 to 100,000 tokens, trained specifically on code corpora. The CodeLlama tokenizer uses a vocabulary of 32,000 tokens trained on a code-heavy corpus. StarCoder uses a vocabulary of 49,152 tokens with reserved space for future tokens. GPT-4's tokenizer, which covers both natural language and code, uses approximately 100,000 tokens.

A key tradeoff specific to code is that a vocabulary trained primarily on natural language handles code poorly. Common code patterns like import numpy as np, def __init__(self):, and return None may be split into many tokens when using a natural language tokenizer, while a code-tuned tokenizer would represent each as one or two tokens. CodeLlama's experiments showed that code-specific tokenization reduced effective sequence lengths by roughly 30 to 40 percent compared to using the Llama-2 natural language tokenizer on identical code. This matters because a shorter sequence allows the model to attend to more context with the same memory budget, which directly translates into the model being able to "see" more of the file it is working with.

Most modern code models use byte-level BPE, which begins by treating each byte (0 through 255) as a base token before learning merge operations. This ensures that no input is ever truly out-of-vocabulary: any byte sequence, including unusual Unicode in string literals, emoji in comments, or non-ASCII identifier characters in international codebases, can be represented.

Byte-level tokenization is especially important for code because source files regularly contain string literals with arbitrary Unicode content, regular expressions with unusual character classes, code comments in languages other than English, and hex-encoded data or base64 constants embedded in source. Without byte-level coverage, these would require special handling or would produce unknown tokens that degrade model quality.

The BPE algorithm then learns merges from this byte base, starting with the most frequent pairs of bytes and iteratively combining them into longer tokens. By training on code-specific corpora, the learned merges naturally favor code patterns: common function names, operator sequences, and indentation patterns become single tokens through repeated merging, while less common byte sequences remain as multi-byte representations.

Python is arguably the hardest language to tokenize well, because its syntax depends on precise indentation to define block structure. A function body indented by four spaces and one indented by three spaces are syntactically different programs (the latter is a syntax error). If indentation is tokenized inconsistently, the model must learn to count spaces character by character, which is wasteful and error-prone.

One solution is to include dedented tokens in the vocabulary: a token that represents four spaces, another for eight spaces, another for twelve spaces, and so on. Quoting the SantaCoder paper, using dedicated indentation tokens reduces Python sequence lengths by approximately 15 percent compared to using the standard GPT-2 tokenizer with no special indentation handling. This is a meaningful gain given Python's dominance in code model training data.

The indentation problem also interacts with FIM. When a FIM split falls at a line boundary, the first token of the middle region is often an indentation sequence followed by code. If the tokenizer does not handle indentation efficiently, the beginning of each infill is dominated by whitespace tokens, making it harder for the model to produce the first meaningful token quickly. In practice, this is why models trained with FIM often include indentation-aware tokenization as a paired improvement.

Numeric literals deserve special attention because they appear in code with high frequency and in diverse formats. Consider the range of ways a value like 255 can be written in Python: 255, 0xFF, 0b11111111, 0o377. These are all the same integer, but their character representations are completely different. A tokenizer that handles them inconsistently may learn that 0xFF and 255 are unrelated values.

Most modern code tokenizers handle this pragmatically by including common hexadecimal, binary, and octal patterns as high-frequency merge targets in BPE, so that 0xFF becomes a small number of tokens while preserving the prefix structure. A more principled approach would canonicalize numeric literals during preprocessing, converting all integers to decimal before tokenization. This is used in some research settings but is not standard in production models because it changes the surface form of the code, which may cause issues when the model needs to reconstruct the original representation.

Floating-point literals present a different challenge: 3.14159, 3.14159e0, and 3.14159E+00 are the same value but tokenize differently. Scientific notation in particular is common in machine learning code (learning rates, epsilon values) and numerical computing. A tokenizer trained on sufficient code data will typically learn that 1e-5 is a natural unit and encode it efficiently.