Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Code LLM Training › Implementation: Analyzing Code Tokenization

Michael BrenndoerferJanuary 29, 20263 min read

Language AI HandbookAll chapters

Part 47 of 67Code Generation

Chapter 357Code LLM Training › Implementation: Analyzing Code Tokenization

Reading section 9 of 13 · Implementation: Analyzing Code Tokenization · 3 minSection 9 of 13 · 3 min

Section 9 of 13 · 3 min read

Code LLM Training › Implementation: Analyzing Code TokenizationOne section · Sections in this chapter9 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's examine how tokenization behavior differs for code versus natural language, using the tokenizers library to implement a simple BPE tokenizer trained on our code corpus.

In[14]:

Code

from tokenizers import Tokenizer, decoders, models, pre_tokenizers, trainers


# Build a small BPE tokenizer trained on our code corpus
def train_code_tokenizer(corpus: list[str], vocab_size: int = 500) -> Tokenizer:
    """Train a small BPE tokenizer on the given corpus."""
    tokenizer = Tokenizer(models.BPE(unk_token="[UNK]"))
    tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
    tokenizer.decoder = decoders.ByteLevel()

    trainer = trainers.BpeTrainer(
        vocab_size=vocab_size,
        special_tokens=["[UNK]", PRE_TOKEN, SUF_TOKEN, MID_TOKEN, EOT_TOKEN],
        show_progress=False,
    )
    tokenizer.train_from_iterator(corpus, trainer=trainer)
    return tokenizer


# Train on our small corpus
code_tokenizer = train_code_tokenizer(deduped_corpus, vocab_size=500)

In[15]:

Code

# Compare tokenization of identifier-rich vs plain text
identifiers = [
    "calculateMeanSquaredError",
    "compute_mean",
    "BinaryTree",
    "insert",
    "factorial",
]

python_code_sample = "def calculate_mean_squared_error(predictions, targets):\n    return np.mean((predictions - targets) ** 2)"
plain_text = "The algorithm calculates the average of squared differences between predicted and actual values."

Out[16]:

Console

Token counts per identifier:
  calculateMeanSquaredError           -> 19 tokens: ['c', 'al', 'c', 'u', 'l', 'ate', 'M', 'e', 'an', '[UNK]', '[UNK]', 'u', 'ar', 'e', 'd', 'E', 'r', 'r', 'or']
  compute_mean                        ->  4 tokens: ['c', 'ompute', '_', 'mean']
  BinaryTree                          ->  3 tokens: ['Bin', 'aryTre', 'e']
  insert                              ->  1 tokens: ['insert']
  factorial                           ->  1 tokens: ['factorial']

Python function signature:   62 tokens
Plain text equivalent:       65 tokens
Ratio (code/text):          0.95

The token count per identifier varies with both identifier length and how often it appeared in the training corpus. Common short names like insert or factorial get compacted into fewer tokens because the BPE trainer has seen enough occurrences to merge them. Longer, compound names require more tokens unless they were common in the training data.

In[17]:

Code

# Analyze vocabulary composition: how many tokens are code-specific
vocab = code_tokenizer.get_vocab()
sorted_vocab = sorted(vocab.items(), key=lambda x: x[1])

# Classify tokens heuristically
keyword_tokens = []
identifier_tokens = []
operator_tokens = []
whitespace_tokens = []
other_tokens = []

python_keywords = {
    "def",
    "return",
    "if",
    "else",
    "elif",
    "for",
    "while",
    "import",
    "from",
    "class",
    "in",
    "not",
    "and",
    "or",
    "is",
    "None",
    "True",
    "False",
}

for token, idx in sorted_vocab:
    clean = token.replace("Ġ", " ").strip()
    if clean in python_keywords:
        keyword_tokens.append(token)
    elif clean.replace("_", "").isalpha() and len(clean) > 2:
        identifier_tokens.append(token)
    elif all(c in "+-*/=<>!&|^~%@:.,;()[]{}" for c in clean) and clean:
        operator_tokens.append(token)
    elif not clean or clean.isspace():
        whitespace_tokens.append(token)
    else:
        other_tokens.append(token)

Out[18]:

Console

Vocabulary composition:
  Python keywords:       14 (4.5%)
  Identifier-like:      108 (35.0%)
  Operators/symbols:     33 (10.7%)
  Whitespace:             6 (1.9%)
  Other:                148 (47.9%)
  Total vocabulary:     309

Sample keyword tokens: ['in', 'or', 'def', 'Ġreturn', 'Ġin', 'Ġif', 'ĠNone', 'Ġfor']
Sample identifier tokens: ['val', 'self', 'Ġself', 'arr', 'value', 'Ġarr', 'Ġretu', 'fig']