Tokenizer Training › Saving and Loading Tokenizers
Language AI HandbookAll chapters Tokenizer Training › Saving and Loading TokenizersOne section · Sections in this chapter5 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Once trained, you need to save your tokenizer for later use. The HuggingFace tokenizers library provides multiple saving formats suited to different use cases. Choosing the right format affects how easily you can integrate the tokenizer into your training pipeline, how portable it is across systems, and how well it interoperates with other tools in the HuggingFace ecosystem.
Saving must ensure reproducibility as well as persistence. A tokenizer that produces different outputs on two different machines, or that changes behavior after a library update, is a silent source of bugs that can be extremely difficult to diagnose. The goal of proper saving is to guarantee that the tokenizer produces exactly the same output on every machine, now and in the future.
Saving to JSON
The native format saves the complete tokenizer configuration as a self-contained JSON file. Every component (normalizer, pre-tokenizer, model, post-processor) is serialized into a human-readable format that captures the full configuration without any implicit state.
import os
import tempfile
temp_dir = tempfile.mkdtemp()
tokenizer_path = os.path.join(temp_dir, "my_tokenizer.json")
tokenizer.save(tokenizer_path)Saved tokenizer to: /var/folders/lz/vn3ps0t51nv5q2g7q4kppt1r0000gn/T/tmpnho8a7ag/my_tokenizer.json File size: 17,933 bytes Tokenizer configuration keys: - version - truncation - padding - added_tokens - normalizer - pre_tokenizer - post_processor - decoder - model
The tokenizer serializes to a compact JSON file containing all necessary components. The model key stores vocabulary and merge rules. The normalizer, pre_tokenizer, and post_processor keys store the processing pipeline configuration. This self-contained file enables exact reproduction of the tokenizer on any system without version-dependent state.
The JSON format is also useful for inspection and debugging. Because the file is human-readable, you can open it and verify that your normalizer is configured correctly, that your vocabulary contains the expected tokens, and that your special token IDs match what your model expects. When tokenizer bugs arise, this is often the first place to look.
Loading a Saved Tokenizer
Loading from JSON is straightforward and produces an exact copy of the original tokenizer:
loaded_tokenizer = Tokenizer.from_file(tokenizer_path)
original_output = tokenizer.encode("Test sentence for verification.")
loaded_output = loaded_tokenizer.encode("Test sentence for verification.")Verification that loaded tokenizer matches original: Original tokens: ['[CLS]', 'Ġt', 'es', 't', 'Ġ', 's', 'en', 't', 'en', 'ce', 'Ġfor', 'Ġ', 'ver', 'ific', 'atio', 'n', '.', '[SEP]'] Loaded tokens: ['[CLS]', 'Ġt', 'es', 't', 'Ġ', 's', 'en', 't', 'en', 'ce', 'Ġfor', 'Ġ', 'ver', 'ific', 'atio', 'n', '.', '[SEP]'] Match: True
The loaded tokenizer produces identical output to the original, confirming that all vocabulary entries and configuration were preserved exactly. This reproducibility is essential for production deployments where tokenizers are saved once and loaded many times across different machines. The key insight is that the JSON format captures the complete algorithmic state, including more than the vocabulary. If you load the tokenizer on a different machine with a different version of the library, you will get the same output as long as the library can parse the JSON format.
Saving for Transformers Integration
To use your tokenizer with the HuggingFace Transformers library, wrap it in PreTrainedTokenizerFast. This wrapper adds the metadata and interface that Transformers models expect, including padding behavior, truncation, and batch encoding. It also makes your tokenizer compatible with AutoTokenizer, which is how most downstream code loads tokenizers.
from transformers import PreTrainedTokenizerFast
wrapped_tokenizer = PreTrainedTokenizerFast(
tokenizer_object=tokenizer,
unk_token="[UNK]",
pad_token="[PAD]",
cls_token="[CLS]",
sep_token="[SEP]",
mask_token="[MASK]",
)
transformers_path = os.path.join(temp_dir, "transformers_tokenizer")
wrapped_tokenizer.save_pretrained(transformers_path)Files saved to transformers_tokenizer/: tokenizer.json: 17,933 bytes tokenizer_config.json: 249 bytes
The Transformers format creates multiple files. tokenizer.json contains the full tokenizer configuration in the native format. tokenizer_config.json stores metadata like special token mappings and the tokenizer class name. special_tokens_map.json explicitly lists all special tokens with their string values. This multi-file format is designed for extensibility: models that add special tokens can add them to the special tokens map without modifying the core vocabulary file.
from transformers import AutoTokenizer
reloaded = AutoTokenizer.from_pretrained(transformers_path)Input: 'Testing the reloaded tokenizer.' Token IDs: [2, 35, 49, 26, 39, 53, 88, 18, 21, 7, 56, 10, 91, 17, 73, 15, 32, 57, 6, 3] Tokens: ['[CLS]', 'Ġt', 'es', 't', 'ing', 'Ġthe', 'Ġre', 'l', 'o', 'a', 'de', 'd', 'Ġto', 'k', 'en', 'i', 'z', 'er', '.', '[SEP]']
Loading via AutoTokenizer demonstrates full compatibility with the Transformers ecosystem. The tokenizer now works with any Transformers model that expects the same vocabulary and special token configuration. This matters for model hub compatibility: when you upload a model to the HuggingFace Hub, your tokenizer travels with the model in the Transformers format, and users can load both with a single from_pretrained call.