Causal Language Modeling › The Autoregressive Factorization

Causal language modeling is built on a mathematical framework called the autoregressive factorization. Before diving into the training objective or the loss function, it helps to understand this underlying structure, because it explains both why the approach is tractable and why it naturally produces generative models. The central challenge is assigning probabilities to sequences of arbitrary length. The autoregressive factorization is the field's mathematically exact solution rather than an approximation.

How do you assign a probability to an entire sentence? This is the fundamental question that language models must answer. Given a sequence of tokens $x_1, x_2, \ldots, x_n$, we want to compute $P(x_1, x_2, \ldots, x_n)$, the probability that this particular sequence occurs in natural language.

Consider what this means in practice. With a vocabulary of 50,000 tokens and sequences of length 100, we'd need to assign probabilities to $50000^{100}$ possible sequences. That's more possibilities than atoms in the observable universe, raised to a power larger than the universe's age in seconds. Storing or computing such a distribution directly is impossible.

We need a way to break this intractable joint probability into manageable pieces. Fortunately, probability theory gives us exactly such a tool: the chain rule.

The chain rule states that any joint probability can be decomposed into a product of conditional probabilities. For a sequence, this means:

Let's trace through what each term represents:

1. $P(x_1)$: The probability of the first token appearing at the start of a sequence. This has no conditioning context.
2. $P(x_2 | x_1)$: Given we've seen the first token, what's the probability of the second?
3. $P(x_3 | x_1, x_2)$: Given the first two tokens, what comes third?
4. And so on, until $P(x_n | x_1, \ldots, x_{n-1})$: the final token, conditioned on everything before it.

This telescoping product captures the sequential nature of language. Each new word depends on what came before, exactly matching our intuition about how text is generated.

We can express this factorization more concisely using product notation:

where:

• $x_t$: the token at position $t$ in the sequence
• $x_{<t}$: all tokens before position $t$, that is, the sequence $(x_1, x_2, \ldots, x_{t-1})$
• $P(x_t | x_{<t})$: the conditional probability of token $x_t$ given all preceding tokens

For the first position where $t=1$, we define $x_{<1}$ as empty, so $P(x_1 | x_{<1}) = P(x_1)$.

Each factor $P(x_t | x_{<t})$ is a conditional distribution over the entire vocabulary. Given everything we've seen so far, what's the probability of each possible next token? This is precisely the question a language model learns to answer.

This decomposition is mathematically exact, not an approximation. The chain rule holds for any joint distribution. What makes it practical is that we've transformed an impossible problem (representing $50000^{100}$ probabilities) into a tractable one (learning a function that outputs a distribution over 50,000 tokens given any context).

The key insight is that while contexts vary enormously, they share statistical patterns. The word "the" tends to be followed by nouns. Questions end with question marks. Technical documents use technical vocabulary. A neural network can learn these patterns and generalize them to new contexts it has never seen before.

There is also a deep connection between this factorization and causality, which is where the name "causal" language modeling comes from. When a model predicts $P(x_t | x_{<t})$, it is computing what should come next based solely on the causal history, everything that appeared before. No information from the future leaks into the prediction. This mirrors real-world language generation: when a human writes a sentence, the word they choose at each moment is influenced by what they have already written, not by what they have not yet written. CLM formalizes this intuitive notion and enforces it architecturally through causal masking, which we'll cover later in the chapter.

Notice that the factorization also implies a natural sampling procedure. To generate a sequence, you sample $x_1$ from $P(x_1)$, then sample $x_2$ from $P(x_2 | x_1)$, then $x_3$ from $P(x_3 | x_1, x_2)$, and so on. At each step you need only the distribution over the next single token, which the model can compute efficiently. This is why the same model used for training, which minimizes prediction loss, can be used directly for generation without any modifications.