mT5 › Multilingual Tokenization
Language AI HandbookAll chapters mT5 › Multilingual TokenizationOne section · Sections in this chapter4 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Training a single model on 101 languages requires a vocabulary that can effectively tokenize all of them. This presents a challenging problem combining computational linguistics and machine learning efficiency. As we covered in the SentencePiece chapter, subword tokenization algorithms learn vocabularies from data by identifying frequently occurring character sequences. The challenge for multilingual models is that vocabulary slots are finite. A larger vocabulary means more parameters in the embedding layer and slower softmax computation during training and inference. Yet a vocabulary that is too small cannot adequately represent the diverse morphological patterns and writing systems found across 101 languages.
mT5 uses a SentencePiece unigram model with a vocabulary of 250,000 subword tokens, compared to T5's 32,000 tokens for English only. This 8x increase accommodates the diverse character sets and morphological patterns across 101 languages. The expansion is necessary because different language families have fundamentally different word formation rules: agglutinative languages like Turkish build complex words by chaining morphemes, while isolating languages like Chinese use single characters to represent concepts. A vocabulary optimized for English would fragment Turkish words into unrecognizable pieces while failing to provide useful decompositions for Chinese characters.
The vocabulary training process samples from mC4 using the same temperature-based sampling as model training. This design choice is important for so low-resource languages contribute meaningful vocabulary entries rather than being drowned out by English. Without temperature sampling during vocabulary construction, the tokenizer would learn subword patterns primarily from English text, leading to poor tokenization quality for low-resource languages.
# Conceptual vocabulary training with temperature sampling
# (Actual mT5 used proprietary SentencePiece training)
def sample_training_text(corpus_sizes, temperature, total_chars=10_000_000):
"""
Sample text for vocabulary training with temperature.
Returns approximate character counts per language.
"""
probs = corpus_sizes ** (1 / temperature)
probs = probs / probs.sum()
# Allocate characters proportionally to tempered probabilities
char_counts = (probs * total_chars).astype(int)
return char_counts
# Compare vocabulary training samples
languages_vocab = ["English", "Russian", "Japanese", "Swahili", "Yoruba"]
sizes_vocab = np.array([2749, 743, 261, 1.8, 0.06])
# Calculate allocations for both sampling strategies
proportional = sample_training_text(sizes_vocab, temperature=1)
tempered = sample_training_text(sizes_vocab, temperature=100)Character allocation for vocabulary training (10M total): Language Proportional T=100 Boost ------------------------------------------------------- English 7,321,178 2,087,125 0.3x Russian 1,978,768 2,059,997 1.0x Japanese 695,099 2,038,559 2.9x Swahili 4,793 1,939,588 404.7x Yoruba 159 1,874,728 11790.7x
Temperature sampling ensures that even low-resource languages contribute substantial training data for vocabulary learning. Without this adjustment, languages like Yoruba might have fewer than 200 characters in the training sample. This is far too few for meaningful subword discovery. The SentencePiece algorithm needs sufficient examples of each language to identify common character patterns and build effective subword units.
Script Coverage
The 250K vocabulary must cover many writing systems. These include Latin and Cyrillic, Arabic and Hebrew, plus Chinese and Japanese. Korean and numerous other writing systems also need representation. Each writing system brings its own characteristics: alphabetic scripts like Latin and Cyrillic build words from individual letters, syllabic scripts like Japanese hiragana represent syllables, and logographic scripts like Chinese use characters that represent morphemes or words. The vocabulary breakdown reflects this diversity, with capacity allocated across script families to ensure adequate coverage:
The vocabulary expansion from 32K to 250K tokens has measurable implications for model efficiency. Each token requires an embedding vector that maps the discrete token to a continuous representation the model can process. The embedding matrix size therefore grows proportionally with vocabulary size, creating a direct tradeoff between linguistic coverage and parameter efficiency:
# Compare embedding matrix sizes
embedding_dim = 1024 # mT5-Large uses 1024-dimensional embeddings
t5_vocab = 32_000
mt5_vocab = 250_000
t5_params = t5_vocab * embedding_dim
mt5_params = mt5_vocab * embedding_dimT5 embedding parameters: 32,768,000 (32.8M) mT5 embedding parameters: 256,000,000 (256.0M) Increase factor: 7.81x
The embedding layer expansion from 32.8M to 256M parameters represents a significant overhead, particularly for smaller model variants. For mT5-Small with approximately 300M total parameters, the embeddings alone account for a substantial fraction of model capacity. This means that a non-trivial portion of the model's learning capacity is dedicated purely to representing the expanded vocabulary, leaving less capacity for learning language understanding and generation. The designers of mT5 judged this tradeoff worthwhile because adequate vocabulary coverage is foundational. A model cannot learn patterns in text it cannot properly tokenize.
Tokenization Efficiency
Multilingual tokenizers face a fertility tradeoff. Tokens optimized for one language may fragment words in another, leading to longer sequences and slower processing. This phenomenon occurs because subword patterns that are common in one language may be rare or nonexistent in another. For instance, the English suffix "-tion" appears frequently and would likely become a single token. However, this character sequence rarely occurs in Japanese. When mT5's tokenizer encounters Japanese text, it must use different subword patterns entirely. Let's examine how mT5's tokenizer handles different languages:
from transformers import T5Tokenizer
# Load mT5 tokenizer (use slow tokenizer to avoid conversion issues)
tokenizer = T5Tokenizer.from_pretrained("google/mt5-small")
# Test sentences (translations of "The quick brown fox jumps over the lazy dog")
test_sentences = {
"English": "The quick brown fox jumps over the lazy dog.",
"Spanish": "El rápido zorro marrón salta sobre el perro perezoso.",
"German": "Der schnelle braune Fuchs springt über den faulen Hund.",
"Russian": "Быстрая коричневая лиса перепрыгивает через ленивую собаку.",
"Japanese": "素早い茶色の狐が怠惰な犬を飛び越える。",
"Chinese": "敏捷的棕色狐狸跳过懒狗。",
"Arabic": "الثعلب البني السريع يقفز فوق الكلب الكسول.",
"Swahili": "Mbweha mwepesi wa kahawia anaruka juu ya mbwa mvivu.",
}mT5 Tokenization Efficiency by Language: Language Chars Tokens Chars/Tok ------------------------------------------ English 44 14 3.14 Spanish 53 19 2.79 German 55 19 2.89 Russian 59 20 2.95 Japanese 19 17 1.12 Chinese 12 14 0.86 Arabic 42 17 2.47 Swahili 52 20 2.60 Average - - 2.35
The tokenization efficiency results show that Latin-script languages achieve roughly 4-6 characters per token. This holds for English and Spanish as well as German, while languages like Japanese and Chinese with unique scripts show different patterns. Arabic script languages fall somewhere in between. These efficiency differences directly impact sequence lengths. Less efficient tokenization means longer sequences for the same content, which affects both computational cost and the model's ability to capture long-range dependencies within its context window. A sentence that tokenizes to 10 tokens in English might require 20 tokens in another language, effectively halving the amount of context the model can consider for that language within a fixed context window.
In practice, this fertility gap has downstream consequences for fine-tuning and inference that are easy to miss. When you set a maximum input length of 512 tokens for an mT5 fine-tuning job, a document that fits comfortably in English might get truncated when expressed in a language with higher fertility. This means that languages already disadvantaged by lower pre-training data volumes can also end up with shorter effective context during fine-tuning, compounding the disadvantage. When deploying mT5 for a multilingual application, it is worth measuring the token-to-word ratio for each target language and adjusting the maximum sequence length accordingly, or at least being aware that different languages effectively see different context windows at the same token limit.