mT5 › Worked Example: Temperature Sampling in Action

Let's trace through a concrete numerical example to solidify the temperature sampling concept. Suppose we have a corpus with just three languages: English (1,000 billion tokens), Turkish (10 billion tokens), and Swahili (1 billion tokens). These numbers are round but representative of the order-of-magnitude differences in the real mC4 corpus.

With proportional sampling ($T=1$), the sampling probabilities are:

$$p_\text{EN} = \frac{1000}{1011} \approx 98.9\%, \quad p_\text{TR} = \frac{10}{1011} \approx 1.0\%, \quad p_\text{SW} = \frac{1}{1011} \approx 0.1\%$$

Under this scheme, for every 1000 training steps, Swahili appears roughly once. A typical pre-training run involves hundreds of billions of steps, so Swahili does appear many times in absolute terms, but relative to English it is almost invisible. The model's Swahili representations will be weak.

Now apply temperature sampling with $T=100$. The adjusted corpus sizes are:

$$1000^{1/100} \approx 1.072, \quad 10^{1/100} \approx 1.023, \quad 1^{1/100} = 1.0$$

The new sum is approximately $1.072 + 1.023 + 1.0 = 3.095$, giving:

$$p_\text{EN}^{(100)} \approx \frac{1.072}{3.095} \approx 34.6\%, \quad p_\text{TR}^{(100)} \approx \frac{1.023}{3.095} \approx 33.1\%, \quad p_\text{SW}^{(100)} \approx \frac{1.0}{3.095} \approx 32.3\%$$

Swahili's sampling probability jumped from 0.1% to 32.3%, a boost factor of over 300. Turkish went from 1.0% to 33.1%, a 33x boost. English dropped from 98.9% to 34.6%. The distribution is now nearly uniform across the three languages, which is exactly what temperature sampling is designed to achieve at high temperatures.

The practical takeaway is striking: temperature sampling turns a situation where Swahili would be nearly invisible into one where it receives roughly equal training signal to English. The model trained with temperature sampling will develop meaningful Swahili representations, while the proportionally sampled model would produce representations so weak as to be nearly useless for Swahili tasks.