T5 Pre-training › T5 Training Scale

The T5 paper introduced models at five different scales, letting systematic study of how performance varies with model size.

T5 was released in five sizes:

As shown in Table t5-variants, the largest model (T5-11B) achieved state-of-the-art results on numerous benchmarks at the time of release. The size progression allowed researchers to study scaling laws and helped establish that larger models consistently perform better when trained appropriately.

T5 models were trained with the following key hyperparameters:

• Sequence length: 512 tokens for both encoder and decoder
• Batch size: 128 sequences (65,536 tokens per batch for T5-Base)
• Training steps: 1 million steps for the base setup
• Optimizer: Adafactor, a memory-efficient variant of Adam
• Learning rate: Inverse square root decay schedule

The total compute for T5-11B was substantial, requiring multiple TPU pods running for weeks. This level of investment established a new baseline for what's possible with sufficient resources.

With 1 million training steps and batch size 128, the model sees:

$$\text{Total examples} = 1{,}000{,}000 \times 128 = 128{,}000{,}000$$

With 512 tokens per sequence:

$$\text{Total tokens seen} = 128{,}000{,}000 \times 512 \approx 65.5 \text{ billion tokens}$$

This means T5 was trained for less than one epoch over C4, seeing each example on average less than once. This is common in large-scale pre-training: the dataset is so large that even partial coverage gives sufficient training signal.