Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Byte Pair Encoding › BPE in Practice with Hugging Face

Michael BrenndoerferUpdated January 24, 20262 min read

Language AI HandbookAll chapters

Part 5 of 67Subword Tokenization

Chapter 29Byte Pair Encoding › BPE in Practice with Hugging Face

Reading section 10 of 14 · BPE in Practice with Hugging Face · 2 minSection 10 of 14 · 2 min

Section 10 of 14 · 2 min read

Byte Pair Encoding › BPE in Practice with Hugging FaceOne section · Sections in this chapter10 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

You will rarely implement BPE from scratch in production. The Hugging Face tokenizers library provides a fast Rust implementation that handles byte-level BPE, special tokens, and all the edge cases. Here's how to train a tokenizer that matches the GPT-2 architecture:

In[28]:

Code

# uv pip install tokenizers
from tokenizers import Tokenizer
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import ByteLevel
from tokenizers.trainers import BpeTrainer

# Build a byte-level BPE tokenizer
tokenizer = Tokenizer(BPE())
tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=False)
tokenizer.decoder = ByteLevelDecoder()

# Define special tokens and target vocabulary size
trainer = BpeTrainer(
    vocab_size=300,
    special_tokens=["<|endoftext|>"],
    show_progress=False,
)

# Train on our sample corpus
tokenizer.train_from_iterator(training_corpus, trainer=trainer)

Out[29]:

Console

Vocabulary size: 112

Encoding with Hugging Face BPE tokenizer:
-------------------------------------------------------
Input   : 'the cat ran'
Tokens  : ['the', 'Ġcat', 'Ġran']
IDs     : [24, 31, 97]
Decoded : 'the cat ran'

Input   : 'dogs and cats play'
Tokens  : ['dogs', 'Ġand', 'Ġcats', 'Ġplay']
IDs     : [93, 37, 49, 69]
Decoded : 'dogs and cats play'

Input   : 'unrecognized word'
Tokens  : ['un', 're', 'c', 'o', 'g', 'n', 'i', 'ed', 'Ġ', 'w', 'o', 'r', 'd']
IDs     : [38, 44, 2, 13, 6, 12, 8, 55, 21, 19, 13, 15, 3]
Decoded : 'unrecognied word'

The Hugging Face tokenizer produces the same algorithmic behavior as our implementation but with dramatically better performance: it can tokenize hundreds of megabytes per second on a single CPU core, making it practical for preprocessing large training datasets. For training a model on a hundred billion tokens, the difference between a Python tokenizer and the Rust-based implementation can mean the difference between preprocessing taking hours versus weeks.

Notice that the byte-level tokenizer represents some characters with multi-character tokens like "Ġ" (the Ġ prefix marks word-initial tokens). This is the GPT-2 convention: the Ġ character encodes the space that precedes a word, so " cat" becomes "Ġcat" as a single token. The decoder understands this convention and converts Ġ back to a space during decoding. Tokens without the prefix represent subword continuations that do not start at a word boundary.