Code LLM Training › Visualizations

Michael BrenndoerferJanuary 29, 20263 min read
Language AI HandbookAll chapters
Part 47 of 67Code Generation
Chapter 357Code LLM Training › Visualizations
Reading section 10 of 13 · Visualizations · 3 minSection 10 of 13 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 357 of 513

Section 10 of 13 · 3 min read

Code LLM Training › VisualizationsOne section · Sections in this chapter10 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Out[19]:
Visualization
Horizontal bar chart showing token counts for five programming identifiers: calculateMeanSquaredError 19, compute_mean 4, BinaryTree 3, insert 1, and factorial 1.
Token count comparison for five programming identifiers using a BPE tokenizer trained on code. The long compound identifier calculateMeanSquaredError requires 19 tokens in this small demonstration vocabulary, while familiar identifiers such as insert and factorial each require one. Token economy reflects both identifier structure and frequency in the training corpus.
Out[20]:
Visualization
Horizontal bar chart of vocabulary composition by token category in a code-trained BPE tokenizer.
Vocabulary composition by token category for a BPE tokenizer trained on Python code. Identifier-like tokens and other subword patterns dominate the vocabulary, showing how much of a program's character budget is spent on named entities. Python keywords represent a small but important fraction, since they appear often enough to be merged early.
Out[21]:
Visualization
Stacked bar chart showing FIM vs causal LM example proportions at different FIM training rates.
FIM transformation rates showing the proportion of training examples using each format across different FIM rates. At a 50% FIM rate (standard practice), roughly equal numbers of FIM and standard causal LM examples are produced. Higher FIM rates improve infilling capability but reduce the model's exposure to standard forward-completion training, which can degrade prefix-only generation quality.