Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GPT-1 › Pre-Training Objective

Michael BrenndoerferUpdated July 22, 202512 min read

Language AI HandbookAll chapters

Part 28 of 67GPT Architecture

Chapter 197GPT-1 › Pre-Training Objective

Reading section 4 of 11 · Pre-Training Objective · 12 minSection 4 of 11 · 12 min

Section 4 of 11 · 12 min read

GPT-1 › Pre-Training ObjectiveOne section · Sections in this chapter4 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

GPT-1's pre-training objective is straightforward: predict the next token given all previous tokens. This is the standard language modeling objective, also called causal language modeling or autoregressive language modeling. Its elegance lies in how much linguistic knowledge it implicitly requires. To predict the next word in a sentence, the model must understand syntax (subject-verb agreement, grammatical case), semantics (word meaning and contextual sense), discourse (what the passage is about, what entities are in play), and pragmatics (what continuation would be coherent given the genre and register of the text). No single one of these abilities is directly trained: they all emerge as instrumental to the core prediction task.

The language modeling objective also provides very dense training signal. For a sequence of $n$ tokens, the model makes $n - 1$ predictions during the forward pass (each position predicts its successor). This means every token in every training example contributes gradient information to every forward pass. Compare this to supervised learning approaches that might have a single label for an entire document: GPT-1's pre-training extracts far more signal from each training example, making efficient use of the BooksCorpus data.

Another important property of the language modeling objective is its generality. Unlike masked language modeling (used in BERT), which requires specific masking strategies and produces a model optimized for filling in blanks, causal language modeling produces a model optimized for sequentially generating or continuing text. This generality is part of why the GPT architecture scaled so effectively: the training objective remained valid and informative regardless of how large the model became or how much data it trained on. There was no need to change the training procedure as the model grew from 117M to 1.5B to 175B parameters.

Language Modeling Objective

Given a sequence of tokens $(x_1, x_2, \ldots, x_n)$ , the model learns to maximize the likelihood of each token given its prefix: $\sum_{i=1}^{n} \log P(x_i | x_1, \ldots, x_{i-1})$ .

For a sequence of tokens $\mathbf{x} = (x_1, x_2, \ldots, x_n)$ , the pre-training objective is to maximize:

\mathcal{L}_{\text{LM}}(\mathbf{x}) = \sum_{i=1}^{n} \log P(x_i | x_1, x_2, \ldots, x_{i-1}; \Theta)

where:

$\mathcal{L}_{\text{LM}}$ : the language modeling loss function (log-likelihood to maximize)
$\mathbf{x}$ : the input sequence of $n$ tokens
$x_i$ : the token at position $i$
$P(x_i | x_1, \ldots, x_{i-1}; \Theta)$ : the probability of token $x_i$ given all preceding tokens, parameterized by model weights $\Theta$
$\Theta$ : all learnable parameters of the model (embeddings, attention weights, feed-forward weights)

In practice, this is implemented as cross-entropy loss between the model's predicted distribution over the vocabulary and the actual next token at each position:

In[16]:

Code

import torch.nn.functional as F


def compute_lm_loss(
    model: GPT1Model,
    input_ids: torch.Tensor,
) -> torch.Tensor:
    """
    Compute the language modeling loss.

    The model predicts the next token at each position,
    and we compute cross-entropy against the actual next token.
    """
    # Get logits from model
    outputs = model(input_ids)
    logits = outputs["logits"]

    # Shift: logits[i] predicts input_ids[i+1]
    shift_logits = logits[:, :-1, :].contiguous()
    shift_labels = input_ids[:, 1:].contiguous()

    # Flatten and compute cross-entropy
    loss = F.cross_entropy(
        shift_logits.view(-1, shift_logits.size(-1)),
        shift_labels.view(-1),
    )

    return loss

Out[17]:

Console

Batch size: 4, Sequence length: 64
Language modeling loss: 312.4927
Perplexity: inf

The loss on random input is approximately $\log(40000) \approx 10.6$ , which corresponds to perplexity equal to the vocabulary size. This is expected for an untrained model that assigns roughly uniform probability across the vocabulary. Training drives this loss down as the model learns to predict more accurately.

Out[18]:

Visualization

Dual-axis line plot showing loss decreasing and perplexity decreasing during training. — Relationship between cross-entropy loss and perplexity during language model training. Perplexity = exp(loss) represents the effective vocabulary size the model is choosing from. An untrained model on a 40K vocabulary has perplexity around 40K (loss around 10.6). A well-trained model typically achieves perplexity of 20-30 on held-out text.

Pre-Training DataLink Copied

GPT-1 was pre-trained on the BooksCorpus dataset, which contains approximately 7,000 unpublished books totaling about 800 million words. The BooksCorpus was chosen for its long-range coherent text, spanning many pages of continuous narrative. This differs from datasets like Wikipedia, where articles are relatively short and self-contained. In Wikipedia, the maximum coherent narrative length is typically a few paragraphs within a section. In BooksCorpus, the model needed to track characters and plotlines while following themes across thousands of tokens, which is precisely the kind of long-range dependency that makes the learned representations useful for downstream tasks.

The choice to use unpublished books (as opposed to published books) was partly motivated by licensing considerations: unpublished books posted to sites like Smashwords were more freely available. The genre distribution skewed toward romance and fiction, which introduced certain biases into the learned representations. For instance, the model learned more about interpersonal relationships and emotions, including informal dialogue, than it would have from a more balanced corpus. Subsequent models like GPT-2 and GPT-3 used more diverse web-scraped data to address this distribution problem, producing more balanced and general-purpose representations.

The BPE tokenizer trained on the BooksCorpus uses 40,000 merge operations, producing a vocabulary that balances coverage and sequence length. Common words like "the" and "cat" receive single tokens, while rare or morphologically complex words are split into subwords. For instance, "revolutionized" might become "revolution" + "ized", and "transformer" might be a single token if it appears frequently enough in the training data. This subword tokenization allows the model to handle any input text, including words not seen during training, by decomposing them into known subword units. The 40,000 merge vocabulary was chosen as a reasonable tradeoff: larger vocabularies reduce average sequence length but require more embedding parameters and make the output projection harder to learn.

Key characteristics of the pre-training setup:

Dataset: BooksCorpus (~800M words from ~7,000 books)
Tokenization: Byte Pair Encoding (BPE) with 40,000 merge operations
Context window: 512 tokens per training example
Batch size: 64 sequences
Training steps: 100 epochs over the dataset
Optimizer: Adam with learning rate 2.5e-4, linear warmup over 2,000 steps, cosine annealing

The choice of BooksCorpus enabled the model to learn long-range dependencies across paragraphs and chapters. This was critical for the transfer learning hypothesis: if the model learned to predict coherent narratives, it might develop representations useful for understanding meaning, not just local syntax.

Out[19]:

Visualization

Horizontal bar chart showing token counts per word for a sample sentence. — BPE tokenization of sample text showing how words are split into subwords. Common words like 'the' remain whole, while rare words like 'transformer' may be split. The token count per word varies based on word frequency in the training corpus. BPE balances vocabulary size against sequence length.

Visualizing Pre-Training DynamicsLink Copied

Let's visualize how next-token prediction works during pre-training. At each position, the model outputs a probability distribution over the vocabulary, and training pushes this distribution toward placing high probability on the actual next token:

Out[20]:

Visualization

Diagram showing input tokens flowing through the model to produce probability distributions for next token prediction. — Next-token prediction during pre-training. The model processes each token position and predicts a distribution over the vocabulary. Cross-entropy loss measures how well the predicted distribution matches the actual next token. Through training, the model learns to place higher probability on correct continuations.

The key insight is that this objective provides dense supervision: every position in every training sequence contributes a gradient signal. Unlike masked language modeling (used in BERT), where only 15% of tokens are predicted, causal language modeling uses every token as a training signal. This makes efficient use of the training data.

Out[21]:

Visualization

Bar chart showing token probabilities with Paris having the highest probability. — Next-token probability distribution from a pre-trained GPT-2 model. Given the context 'The capital of France is', the model assigns high probability to 'Paris' and related tokens. The long tail shows the model maintains probability mass across many plausible continuations, with most tokens receiving near-zero probability.