Decoder Architecture › Understanding Autoregressive Generation

Before diving into the architecture, let's clarify what autoregressive generation means in practice. The decoder generates text one token at a time, where each new token is conditioned on all previously generated tokens.

Given a prompt "The cat sat on the", generation proceeds as follows:

1. Process the prompt through the decoder to get a representation for each position
2. Use the representation at the final position to predict the next token: "mat"
3. Append "mat" to the sequence and process again
4. Use the new final position to predict the next token: "and"
5. Continue until generating a stop token or reaching a length limit

Mathematically, we're factorizing the joint probability of a sequence into a product of conditional probabilities. This is known as the chain rule of probability, and it expresses the idea that generating a sequence is equivalent to making a series of next-token predictions:

$$P(x_1, x_2, \ldots, x_T) = \prod_{t=1}^{T} P(x_t | x_1, x_2, \ldots, x_{t-1})$$

where:

• $P(x_1, x_2, \ldots, x_T)$: the joint probability of the entire sequence of $T$ tokens
• $x_t$: the token at position $t$ in the sequence
• $T$: the total sequence length
• $\prod_{t=1}^{T}$: the product over all positions from 1 to $T$
• $P(x_t | x_1, \ldots, x_{t-1})$: the conditional probability of token $x_t$ given all previous tokens

Each factor in this product represents one prediction step. The factor $P(x_1)$ predicts the first token with no context. The factor $P(x_2 | x_1)$ predicts the second token given the first. By the time we reach $P(x_T | x_1, \ldots, x_{T-1})$, we're predicting the final token given the entire preceding context.

The decoder computes each of these conditional probabilities. It takes the sequence so far, applies its transformer blocks to build contextual representations, and outputs a probability distribution over the vocabulary for the next token.