Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

mT5 › Temperature-Based Language Sampling

Michael BrenndoerferOctober 20, 20258 min read

Language AI HandbookAll chapters

Part 25 of 67Encoder-Decoder Models

Chapter 182mT5 › Temperature-Based Language Sampling

Reading section 3 of 11 · Temperature-Based Language Sampling · 8 minSection 3 of 11 · 8 min

Section 3 of 11 · 8 min read

mT5 › Temperature-Based Language SamplingOne section · Sections in this chapter3 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The large disparity in corpus sizes across languages presents a basic challenge for multilingual model training. If we simply train on data in proportion to how much exists, English would dominate the training process. The model would see English text roughly 45,000 times more often than Yoruba text. Such a model might achieve excellent English performance, but would perform poorly for speakers of low-resource languages. On the other hand, if we sample equally from all languages, we would cycle through the entire Yoruba corpus thousands of times while barely scratching the surface of available English data. This leads to severe overfitting on low-resource languages and underutilization of high-resource ones.

To address the resource imbalance, mT5 uses temperature-based sampling that interpolates between proportional and uniform sampling. This approach allows practitioners to control the tradeoff between respecting natural data proportions and so adequate representation for all languages. Let $p_l$ represent the probability of sampling from language $l$ during training. If we sample proportionally to corpus size, we have:

p_l = \frac{|D_l|}{\sum_{l'} |D_{l'}|}

where:

$p_l$ : the probability of sampling from language $l$ during training
$|D_l|$ : the number of tokens in language $l$ 's subcorpus
$\sum_{l'} |D_{l'}|$ : the total number of tokens across all languages, serving as a normalizing constant

This straightforward proportional approach would give English roughly 27% of all samples while many languages would appear in less than 0.01% of batches, essentially invisible during training.

The key insight behind temperature sampling is that we can systematically compress the differences between corpus sizes by applying a mathematical transformation. Temperature sampling modifies these probabilities by raising them to a power $1/T$ , where $T$ is the temperature. The intuition is that the exponent $1/T$ acts as a "flattening" operator on the distribution. When we raise numbers of vastly different magnitudes to a small power, their differences shrink dramatically. Consider what happens when you raise both 1,000,000 and 1 to the power 0.01: you get approximately 1.15 and 1.0 respectively. The millionfold difference has compressed to just 15%. This compression is precisely what temperature sampling exploits:

p_l^{(T)} = \frac{|D_l|^{1/T}}{\sum_{l'} |D_{l'}|^{1/T}}

where:

$p_l^{(T)}$ : the temperature-adjusted sampling probability for language $l$
$T$ : the temperature parameter controlling the balance between proportional and uniform sampling
$|D_l|^{1/T}$ : the corpus size raised to power $1/T$ , which compresses differences between languages as $T$ increases
$\sum_{l'} |D_{l'}|^{1/T}$ : the sum of adjusted corpus sizes across all languages, so probabilities sum to 1

The key insight is that as temperature $T$ increases, the exponent $1/T$ approaches zero, making $|D_l|^{1/T}$ approach 1 for all languages regardless of their original corpus size. This progressively flattens the distribution toward uniform sampling. Understanding this behavior requires thinking carefully about what happens to the exponentiation as the temperature parameter changes.

To see why this works mathematically, consider two extreme cases:

When $T = 1$ : We have $|D_l|^{1/1} = |D_l|$ , so probabilities are exactly proportional to corpus size
When $T \to \infty$ : We have $|D_l|^{1/\infty} = |D_l|^{0} = 1$ for all languages, giving uniform probabilities of $1/L$ where $L$ is the number of languages

The benefit of this approach becomes clear when we consider intermediate temperatures. Rather than requiring a separate mechanism to interpolate between proportional and uniform sampling, the temperature parameter provides a continuous dial that smoothly transitions between these extremes. For example, with English at 2749B tokens and Yoruba at 0.06B tokens, at $T=1$ the ratio is 45,817:1, but at $T=100$ the ratio becomes $(2749)^{0.01}:(0.06)^{0.01} \approx 1.08:0.97 \approx 1.1{:}1$ , a significant compression. This means that even with temperature sampling, high-resource languages still receive slightly more training signal. This reflects their richer and more diverse content, but the gap narrows enough that low-resource languages can learn meaningful representations.

Temperature Parameter

Temperature $T$ controls interpolation between sampling strategies. At $T=1$ , sampling is proportional to corpus size. As $T \to \infty$ , sampling approaches uniform across languages. The mT5 authors found $T=100$ to work well, significantly boosting low-resource languages while still favoring high-resource languages.

Historical Context: Borrowing from Multilingual MT

Temperature-based sampling for multilingual training was not invented by the mT5 team. Earlier multilingual machine translation systems, including Google's massively multilingual neural MT work from 2019, faced the same resource imbalance problem and adopted similar sampling strategies. The mT5 contribution was showing that this technique transfers cleanly to encoder-decoder pre-training and produces cross-lingual representations powerful enough for zero-shot transfer across diverse downstream tasks, not just translation.

Let's see how temperature affects sampling probabilities:

In[4]:

Code

import numpy as np

# Corpus sizes for selected languages (in billions of tokens)
languages = [
    "English",
    "Russian",
    "Spanish",
    "Japanese",
    "Swahili",
    "Telugu",
    "Yoruba",
]
corpus_sizes = np.array([2749, 743, 416, 261, 1.8, 1.2, 0.06])


def temperature_sampling(sizes, temperature):
    """Compute sampling probabilities with temperature."""
    # Raise to power 1/T
    adjusted = sizes ** (1 / temperature)
    # Normalize to probabilities
    return adjusted / adjusted.sum()


# Compare different temperatures
temperatures = [1, 10, 100, float("inf")]

# Calculate probabilities for each temperature
results = {}
for t in temperatures:
    if t == float("inf"):
        results[t] = np.ones(len(languages)) / len(languages)
    else:
        results[t] = temperature_sampling(corpus_sizes, t)

Out[5]:

Console

Sampling probabilities at different temperatures:

Language             T=1        T=10       T=100         T=∞
------------------------------------------------------------
English         65.8907     20.9243     14.9296     14.2857 
Russian         17.8089     18.3583     14.7355     14.2857 
Spanish          9.9711     17.3238     14.6503     14.2857 
Japanese         6.2559     16.5347     14.5822     14.2857 
Swahili          0.0431     10.0522     13.8742     14.2857 
Telugu           0.0288      9.6528     13.8181     14.2857 
Yoruba           0.0014      7.1540     13.4102     14.2857 

------------------------------------------------------------
Total             100.0       100.0       100.0       100.0

The table reveals the substantial effect of temperature on sampling distribution. At $T=1$ (proportional sampling), English would comprise over 65% of training data while Yoruba gets essentially zero. At $T=100$ (used by mT5), English is reduced to about 4.5% while Yoruba increases to 2.5%. This represents a large boost for low-resource languages. Yoruba's sampling probability increases by a factor of over 1000. The practical consequence is important: a model trained with proportional sampling would see Yoruba text so rarely that it could never learn the language's patterns, while temperature sampling ensures Yoruba appears frequently enough to develop useful representations of the language.

Out[6]:

Visualization

Line plot showing how sampling probabilities change across temperatures for different languages. — Effect of temperature on language sampling probabilities. Higher temperatures flatten the distribution, giving low-resource languages more representation during training.

Line plot showing the boost factor for Yoruba as temperature increases. — Yoruba boost factor across temperatures. Higher temperatures disproportionately increase sampling probability for low-resource languages.

The temperature-based approach allows low-resource languages to receive meaningful training signal without completely ignoring the abundant high-resource data. However, this comes with a tradeoff: high-resource language performance slightly decreases compared to what could be achieved with proportional sampling. The mT5 authors found $T=100$ provided a good balance between these competing objectives. This choice reflects careful empirical tuning. Lower temperatures would still underrepresent low-resource languages, while higher temperatures would waste the rich diversity of high-resource data by undersampling it.

The code demonstrates that at $T=100$ , English's sampling probability drops from over 65% to around 14%, while Yoruba increases from nearly 0% to about 2%. This represents a boost factor of over 1000x for the low-resource language.

In practice, temperature sampling is applied both during model pre-training and during vocabulary training. Applying it only during model training but not vocabulary training would create a mismatch: the tokenizer would learn subword units primarily from English text. This produces a vocabulary ill-suited to the languages that the model then trains on more heavily. The mT5 team applied the same $T=100$ schedule to both stages. This keeps the vocabulary reflects the same language balance that the model itself encounters during pre-training. This consistency is easy to overlook but important for low-resource language quality.