GPT-1 › Key Parameters

When working with GPT-1-era models for transfer learning, these parameters have the greatest impact on performance. Understanding why each parameter matters requires thinking about the two-phase nature of the approach: the pre-trained model is a carefully optimized starting point, and fine-tuning is a controlled perturbation of that starting point toward a task-specific objective. The goal is to move far enough to specialize for the task but not so far as to destroy the general-purpose representations.

• Learning rate: Fine-tuning typically uses 1-2 orders of magnitude lower learning rate than pre-training (e.g., 2-6e-5 vs 2.5e-4). Higher rates risk catastrophic forgetting of pre-trained knowledge. The intuition is that the pre-trained model already resides in a good region of parameter space; large gradient steps would carry it far from that region before the task-specific signal can guide it back to a good local optimum.

• Epochs: 2-4 epochs usually suffice for fine-tuning. Unlike training from scratch, the model starts from a strong initialization and quickly adapts to the task. Training for more epochs tends to overfit the fine-tuning data, especially for small datasets, as the model begins to memorize training examples rather than generalize.

• Batch size: 16-32 is typical for fine-tuning. Larger batches can speed training but may require learning rate adjustment. For very small fine-tuning datasets (under 1,000 examples), smaller batches with more gradient steps may help by providing more frequent parameter updates.

• Auxiliary LM weight ($\lambda$): 0.5 as recommended in the paper, but can be tuned per task. Higher values provide more regularization, useful for smaller datasets. On very large fine-tuning datasets (100K+ examples), you can often reduce $\lambda$ or set it to zero, since the abundant labeled data reduces the risk of overfitting.

• Dropout: 0.1 in attention and feed-forward layers. Can be increased for very small fine-tuning datasets to prevent overfitting. The interaction between dropout and auxiliary LM loss means that both regularization mechanisms together can be more effective than either alone.

• Layer selection: For feature extraction (frozen model), intermediate layers (6-9 for a 12-layer model) often outperform the final layer, which becomes specialized for next-token prediction. The final layer's representations are optimized for language modeling, which may not align with the feature requirements of your downstream task. Intermediate layers retain more general-purpose linguistic structure.

• Warmup: Linear warmup over 0.2% of training steps helps stabilize early training when fine-tuning. Warmup prevents large early gradient steps (which can destabilize the pre-trained representations) by gradually increasing the learning rate from a near-zero value to the target rate over the first few hundred steps.