Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Causal Language Modeling › From Sequence to Training Examples

Michael BrenndoerferUpdated July 7, 20253 min read

Language AI HandbookAll chapters

Part 22 of 67Pre-training Objectives

Chapter 155Causal Language Modeling › From Sequence to Training Examples

Reading section 4 of 13 · From Sequence to Training Examples · 3 minSection 4 of 13 · 3 min

Section 4 of 13 · 3 min read

Causal Language Modeling › From Sequence to Training ExamplesOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A key insight of CLM is that a single sequence yields multiple training examples. For a sequence of length $n$ , we predict tokens at positions 2 through $n$ using contexts of increasing length. This is what makes CLM so data-efficient compared to other supervised learning tasks. In image classification, a single image is a single training example. In CLM, a single sentence of 20 words provides 19 training examples, and a document of 2,000 tokens provides 1,999. The same bytes of raw text generate orders of magnitude more gradient signal than would be possible with explicit labeling.

Consider the sentence "The cat sat on the mat". We create training pairs:

Training pairs from the sentence "The cat sat on the mat", showing how each position predicts the next token.
Context	Target
[START]	The
[START] The	cat
[START] The cat	sat
[START] The cat sat	on
[START] The cat sat on	the
[START] The cat sat on the	mat

The model sees the same sequence but learns from every position simultaneously. This is implemented using a clever shifting trick:

In[8]:

Code

def prepare_clm_batch(token_ids):
    """
    Prepare input and target tensors for CLM training.

    Input tokens are [0:n-1], targets are [1:n].
    The model predicts each token given all previous tokens.
    """
    # Input: all tokens except the last
    input_ids = token_ids[:, :-1]

    # Target: all tokens except the first (shifted by 1)
    labels = token_ids[:, 1:]

    return input_ids, labels


# Example: tokenized sequence
tokens = torch.tensor(
    [[101, 2054, 3921, 2068, 1999, 2023, 102]]
)  # [CLS] ... [SEP]
inputs, targets = prepare_clm_batch(tokens)

Out[9]:

Console

Original sequence: [101, 2054, 3921, 2068, 1999, 2023, 102]
Input tokens:      [101, 2054, 3921, 2068, 1999, 2023]
Target tokens:     [2054, 3921, 2068, 1999, 2023, 102]

Training pairs:
  Context: [101] → Target: 2054
  Context: [101, 2054] → Target: 3921
  Context: [101, 2054, 3921] → Target: 2068
  Context: [101, 2054, 3921, 2068] → Target: 1999
  Context: [101, 2054, 3921, 2068, 1999] → Target: 2023
  Context: [101, 2054, 3921, 2068, 1999, 2023] → Target: 102

The inputs and targets are offset by one position. At position $i$ , the model receives tokens $0, 1, \ldots, i$ as input and predicts token $i+1$ . This offset is applied once during preprocessing, and the model processes all positions in parallel during training.

In practice, this parallel processing is why transformers are so much faster to train than recurrent neural networks. An RNN would process the sequence step by step, computing each hidden state sequentially before moving to the next. A transformer with causal masking computes all hidden states in a single forward pass, using the mask to enforce the left-to-right constraint. This enables much larger batch sizes and dramatically higher GPU utilization.