RoBERTa › Key Parameters

When implementing or fine-tuning RoBERTa, these parameters most significantly impact performance. For fine-tuning specifically, the hyperparameter choices can have a large effect on final task performance, sometimes larger than the choice between BERT and RoBERTa. The RoBERTa paper recommends tuning learning rate, batch size, and number of training epochs on a held-out validation set, using a small sweep over reasonable ranges.

• mask_prob (default: 0.15): Fraction of tokens to mask per sequence. The 15% rate balances training signal strength against context preservation. Higher rates mask more tokens per update but risk destroying too much context for accurate predictions.

• batch_size (RoBERTa default: 8192): Number of sequences per training step. Larger batches provide more stable gradient estimates, particularly important for MLM where only 15% of tokens contribute gradients. Requires proportional learning rate scaling.

• max_seq_len (default: 512): Maximum sequence length. Longer sequences capture more context but require quadratically more memory for attention. RoBERTa packs contiguous text up to this limit.

• vocab_size (RoBERTa: 50265): Size of the BPE vocabulary. RoBERTa uses a 50K vocabulary trained on its larger dataset, slightly larger than BERT's ~30K.

• special_token_ids: Token IDs that should never be masked, including <s> (CLS), </s> (SEP), and <pad>. These tokens serve structural purposes and masking them would disrupt input format.

• learning_rate: Typically 1e-4 to 6e-4 for pretraining. When scaling batch size, apply the linear scaling rule: if you double batch size, double the learning rate (with appropriate warmup). For fine-tuning, typical ranges are 1e-5 to 5e-5, with smaller values generally safer for avoiding catastrophic forgetting of pretrained representations.

• warmup_steps: The number of training steps over which the learning rate is linearly increased from 0 to its peak value. RoBERTa used 24,000 warmup steps out of 500,000 total, roughly 5% of training. For fine-tuning, 6% warmup followed by linear decay is a standard choice. Warmup prevents large destructive updates in the early stages of training when the optimizer state is not yet well-calibrated.

• weight_decay: L2 regularization applied to all parameters except biases and layer normalization weights. RoBERTa used a weight decay of 0.01. This helps prevent overfitting during pretraining and is especially important during fine-tuning on small datasets where the risk of overfitting the task-specific head is high.