BERT Fine-tuning › Key Parameters
Language AI HandbookAll chapters BERT Fine-tuning › Key ParametersOne section · Sections in this chapter10 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The most important hyperparameters for BERT fine-tuning:
-
learning_rate (2e-5 to 5e-5): Controls how quickly the model adapts. Too high causes instability and forgetting; too low learns too slowly. Start with 2e-5 for sensitive tasks, 5e-5 when the task is less sensitive to larger updates.
-
num_epochs (2-4): Number of passes through the training data. More epochs increase forgetting risk. Use early stopping to find the optimal point.
-
batch_size (16-32): Number of examples per gradient update. Larger batches are more stable but require more memory. Gradient accumulation can simulate larger batches.
-
warmup_ratio (0.1): Fraction of training spent warming up the learning rate. Prevents early instability from noisy gradients.
-
weight_decay (0.01): L2 regularization coefficient. Penalizes large weight changes, helping preserve pre-trained knowledge.
-
max_length (128-512): Maximum sequence length after tokenization. Longer sequences require more memory and compute. Use the shortest length that captures your data.
-
dropout (0.1): Applied in the classifier head. Increase for small datasets to reduce overfitting.
-
gradient_clip (1.0): Maximum gradient norm. Prevents exploding gradients during training.