BART Pre-training › Training Dynamics

Michael BrenndoerferOctober 19, 20252 min read
Language AI HandbookAll chapters
Part 25 of 67Encoder-Decoder Models
Chapter 181BART Pre-training › Training Dynamics
Reading section 13 of 16 · Training Dynamics · 2 minSection 13 of 16 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 181 of 513

Section 13 of 16 · 2 min read

BART Pre-training › Training DynamicsOne section · Sections in this chapter13 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

BART's pre-training uses standard practices for large language models, with some choices informed by its architecture:

Optimizer: AdamW with β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, weight decay of 0.01. AdamW properly decouples weight decay from the adaptive learning rate.

Learning rate schedule: Linear warmup for the first 10,000 steps, then linear decay. This follows the pattern we saw in BERT pre-training but with a longer warmup suitable for larger models.

Batch size: Large batches (8,000 tokens per batch for BART-Base) improve training stability and allow higher learning rates.

Data: BART was trained on the same data as RoBERTa: 160GB of books and Wikipedia articles, supplemented with news and other web text. This diverse corpus exposes the model to varied writing styles and domains.

Out[28]:
Visualization
Line plot showing learning rate increasing linearly then decreasing linearly over training steps.
BART learning rate schedule showing linear warmup followed by linear decay. The peak learning rate is reached after warmup steps, then decreases linearly to zero.

The warmup phase prevents early training instability when gradients are noisy and model weights are randomly initialized. The subsequent decay helps the model converge to a stable minimum.