Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GPT-1 › Fine-Tuning Approach

Michael BrenndoerferUpdated July 22, 202510 min read

Language AI HandbookAll chapters

Part 28 of 67GPT Architecture

Chapter 197GPT-1 › Fine-Tuning Approach

Reading section 5 of 11 · Fine-Tuning Approach · 10 minSection 5 of 11 · 10 min

Section 5 of 11 · 10 min read

GPT-1 › Fine-Tuning ApproachOne section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

GPT-1's fine-tuning approach was central to its success. Rather than designing task-specific architectures, the same pre-trained model was adapted to each task with minimal modifications: add a simple classifier head and fine-tune all parameters end-to-end. This simplicity was part of what made the approach so powerful. Previous work on transfer learning in NLP often involved complex procedures: train an ELMo model, freeze its weights, concatenate its outputs with task-specific features, train the task-specific layers. GPT-1 replaced this elaborate pipeline with a clean two-step process: pre-train everything on text, then fine-tune everything on the task.

The key design principle governing fine-tuning is parameter efficiency. Because the pre-trained model already encodes rich linguistic knowledge, the fine-tuning step needs to accomplish very little: it needs to steer the model's existing representations toward task-relevant distinctions, not learn language from scratch. This is why fine-tuning requires only a few epochs and a low learning rate. The model starts from a near-optimal initialization for language understanding tasks, and fine-tuning makes small adjustments to specialize its behavior. A higher learning rate would risk overwriting the pre-trained knowledge, a phenomenon called catastrophic forgetting, where gradient steps driven by task-specific data push the parameters far enough from their pre-trained values that the general language understanding is lost.

The auxiliary language modeling loss during fine-tuning addresses catastrophic forgetting directly. By including the pre-training objective in the fine-tuning loss (with a coefficient $\lambda = 0.5$ ), the model is penalized for drifting too far from its pre-trained behavior. The task-specific labels push the model toward the task, while the language modeling objective acts as an anchor, keeping the representations close to their pre-trained values. This regularization effect is particularly valuable for small fine-tuning datasets, where the task labels alone would quickly overfit the small amount of labeled data.

Input TransformationLink Copied

The central design choice was to transform each task into a format compatible with the language model's interface. All tasks were converted into sequences of tokens that the model processes left-to-right. This is a non-trivial design challenge: different NLP tasks have very different input structures. Sentiment classification takes a single text. Entailment takes two texts with a logical relationship between them. Similarity takes two texts that should be compared. Multiple-choice question answering takes a context passage and several candidate answers. GPT-1 provides a unified solution: represent all inputs as linear sequences with special delimiter tokens marking the boundaries between text segments.

The special tokens, [Start], [Delim], and [Extract], are added to the vocabulary and their embeddings are learned from scratch during fine-tuning. The [Start] token signals the beginning of a new example. The [Delim] token separates two text segments (e.g., premise from hypothesis in entailment, or context from answer in QA). The [Extract] token (sometimes called the classification token) appears at the end of the sequence, and its final hidden-state representation is used as the input to the classification head. Because of causal attention, the [Extract] token's representation aggregates information from all preceding tokens. This provides a summary of the entire input.

Out[22]:

Visualization

Diagram showing how classification, entailment, similarity, and QA tasks are formatted as token sequences. — GPT-1 input transformation for different NLP tasks. Each task is converted into a token sequence with special delimiters. The final token's representation is used for classification. This unified format allows a single model to handle diverse tasks.

The input transformations follow a consistent pattern:

Classification: [Start] text [Extract], where the representation at [Extract] is used for classification
Entailment: [Start] premise [Delim] hypothesis [Extract], which determines if premise entails hypothesis
Similarity: [Start] text1 [Delim] text2 [Extract], processing both orderings and averaging
Multiple Choice/QA: [Start] context [Delim] answer [Extract], scoring each answer independently with softmax over scores

The [Extract] token (sometimes called [CLS] in other models) is a special token whose final representation is used for the classification head. Because of causal attention, this token's representation aggregates information from the entire preceding sequence.

Fine-Tuning LossLink Copied

During fine-tuning, GPT-1 uses a combined objective that includes both the task-specific loss and a language modeling auxiliary loss. The combined loss is computed by adding a fraction of the pre-training language modeling objective to the primary task-specific objective:

\mathcal{L}_{\text{finetune}} = \mathcal{L}_{\text{task}}(y | \mathbf{x}) + \lambda \cdot \mathcal{L}_{\text{LM}}(\mathbf{x})

where:

$\mathcal{L}_{\text{task}}$ : the task-specific loss (e.g., cross-entropy for classification)
$y$ : the ground truth label for the task
$\mathbf{x}$ : the input sequence
$\lambda$ : the auxiliary loss weight (set to 0.5 in the original paper)
$\mathcal{L}_{\text{LM}}$ : the language modeling loss on the input, computed identically to the pre-training objective

Why does this formula make sense? Notice that $\mathcal{L}_{\text{LM}}(\mathbf{x})$ depends on $\mathbf{x}$ but not on $y$ : it measures how well the model can predict the tokens in the input sequence, regardless of the task label. This loss is computable for any text sequence, supervised or not. During fine-tuning, the task loss $\mathcal{L}_{\text{task}}$ pushes the model toward correct task predictions, while $\mathcal{L}_{\text{LM}}$ simultaneously pushes the model to maintain its ability to predict tokens in the input. The coefficient $\lambda = 0.5$ means the language modeling objective receives half the weight of the task objective, strong enough to regularize but not so strong as to prevent task learning.

The auxiliary language modeling loss serves two purposes. First, it acts as a regularizer, preventing the model from forgetting useful language patterns during fine-tuning. A model fine-tuned with only task supervision may rapidly overfit the task-specific data, especially for small datasets, while simultaneously degrading at language modeling. The auxiliary loss prevents this degradation. Second, it provides additional gradient signal, particularly useful when task-specific training data is limited. When only a few hundred labeled examples are available, the language modeling objective on the input tokens provides thousands of additional gradient signals per batch, effectively augmenting the training signal.

In[23]:

Code

class GPT1ForClassification(nn.Module):
    """GPT-1 with classification head for fine-tuning."""

    def __init__(
        self,
        base_model: GPT1Model,
        num_classes: int,
        extract_token_id: int = 40001,  # Special [Extract] token
    ):
        super().__init__()
        self.base_model = base_model
        self.classifier = nn.Linear(768, num_classes)
        self.extract_token_id = extract_token_id

    def forward(
        self,
        input_ids: torch.Tensor,
        labels: torch.Tensor | None = None,
        lm_weight: float = 0.5,
    ) -> dict[str, torch.Tensor]:
        outputs = self.base_model(input_ids)
        logits = outputs["logits"]
        hidden_states = outputs["last_hidden_state"]

        # Find [Extract] token positions (last token in each sequence)
        batch_size, seq_len = input_ids.shape
        extract_positions = (input_ids == self.extract_token_id).float()

        # If no extract token found, use last position
        if extract_positions.sum() == 0:
            extract_hidden = hidden_states[:, -1, :]
        else:
            # Get hidden state at extract token position
            extract_idx = extract_positions.argmax(dim=1)
            extract_hidden = hidden_states[
                torch.arange(batch_size), extract_idx
            ]

        # Classification logits
        class_logits = self.classifier(extract_hidden)

        result = {"class_logits": class_logits, "lm_logits": logits}

        if labels is not None:
            # Task loss (classification)
            task_loss = F.cross_entropy(class_logits, labels)

            # LM loss (auxiliary)
            shift_logits = logits[:, :-1, :].contiguous()
            shift_labels = input_ids[:, 1:].contiguous()
            lm_loss = F.cross_entropy(
                shift_logits.view(-1, shift_logits.size(-1)),
                shift_labels.view(-1),
            )

            # Combined loss
            total_loss = task_loss + lm_weight * lm_loss
            result["loss"] = total_loss
            result["task_loss"] = task_loss
            result["lm_loss"] = lm_loss

        return result

Out[24]:

Console

Classification logits shape: torch.Size([4, 3])
Total loss: 146.2779
Task loss: 1.4628
LM auxiliary loss: 289.6301

The classification head adds minimal parameters (just 768 × num_classes), keeping fine-tuning efficient. The combined loss balances learning the task while maintaining the language model's learned representations.

Fine-Tuning HyperparametersLink Copied

GPT-1 used the following hyperparameters for fine-tuning:

Learning rate: 6.25e-5 (lower than pre-training)
Batch size: 32
Epochs: 3 (most tasks)
LM auxiliary weight ( $\lambda$ ): 0.5
Warmup: Linear warmup over 0.2% of training
Dropout: 0.1 on classifier, 0.1 in attention/residual

The lower learning rate prevents catastrophic forgetting of pre-trained knowledge. Just 3 epochs were typically sufficient because the model started from a strong initialization. This contrasts sharply with training from scratch, which might require hundreds of epochs.

Out[25]:

Visualization

Line plot comparing accuracy curves for pre-trained and randomly initialized models across training epochs. — Fine-tuning performance vs. epochs for pre-trained vs. randomly initialized models. Pre-trained models (GPT-1) reach high accuracy within 2-3 epochs, while random initialization requires many more epochs and achieves lower final performance. This demonstrates the efficiency gains from transfer learning.