T5 Architecture › Encoder-Decoder Architecture

T5 uses the original Transformer's encoder-decoder structure, with modifications that improve training stability and performance. This architectural choice shows an insight about how different NLP tasks process information. Some tasks, like classification or sentiment analysis, primarily require understanding an input. Others, like open-ended writing, primarily require generating new content. But tasks that combine understanding with generation require both: deep comprehension of the input followed by structured generation of output. The encoder-decoder architecture gives dedicated machinery for each phase of this process.

The encoder processes the input sequence bidirectionally. This creates rich contextual representations where each token's embedding shows its relationships with every other token in the input. The decoder then generates output tokens one at a time, attending to both the encoder's output and its own previous predictions. This separation allows the encoder to build a complete understanding of the source text before the decoder begins generating. This keeps even the first output token benefits from full context about the input.

Think of the encoder as a reader who reads the entire source document carefully before picking up a pen, and the decoder as the writer who drafts the output word by word while frequently consulting the reader's notes. The notes (the encoder's hidden states) do not change as the writer works; they stand for a fixed, complete summary of the source. At each step of writing, the decoder can ask: "which parts of the source are most relevant to what I am about to say?" The cross-attention mechanism gives the mechanism for this consultation.

The encoder consists of stacked Transformer blocks, each containing self-attention and feed-forward layers. Unlike GPT-style decoders, the encoder uses bidirectional attention. Every token can attend to every other token in the input, regardless of position. This bidirectionality is important for tasks like translation and summarization, where understanding a word often requires seeing what comes both before and after it. Consider the sentence "The bank was steep." Determining whether "bank" refers to a financial institution or a riverbank requires seeing "steep," which comes later in the sequence. A unidirectional encoder would have already committed to a representation of "bank" before seeing "steep," and correcting that representation would require it to propagate updates backward through attention, which is less direct than bidirectional attention that simply sees the whole sentence at once.

Each encoder block applies a carefully orchestrated sequence of operations that transform token representations while maintaining training stability:

1. Layer normalization (applied before attention, not after)
2. Multi-head self-attention with relative position biases
3. Residual connection
4. Layer normalization
5. Position-wise feed-forward network
6. Residual connection

T5 uses "pre-norm" placement, where layer normalization comes before each sublayer rather than after. This architectural choice, explored systematically in the T5 paper, improves training stability, especially for deeper models. The intuition is straightforward: normalizing inputs to each sublayer ensures that attention and feed-forward operations receive consistently scaled values, preventing the accumulation of extreme activations that can destabilize training in deep networks. In the original Transformer, layer normalization was applied after the residual connection ("post-norm"). Empirically, pre-norm training is more stable and allows higher learning rates, though post-norm sometimes reaches better final performance when the training does converge. T5's systematic experiments confirmed that pre-norm was the better default for large-scale training.

The feed-forward network inside each encoder block follows a specific structure: a linear expansion to a larger dimension (typically 4 times the hidden size), a nonlinear activation, and a linear projection back to the hidden dimension. In T5, this uses a gated activation function called GeGLU (Gated Linear Unit with GELU), which allows the network to selectively pass or gate information based on a learned gating signal. The result is that the feed-forward layer acts somewhat like a key-value memory: certain input patterns activate certain "memory slots" and retrieve associated information. Research on larger models has found that a significant fraction of factual knowledge is stored in the feed-forward layers rather than the attention layers.

The decoder mirrors the encoder's structure but adds cross-attention to incorporate information from the encoded input. This cross-attention mechanism is what allows the decoder to "consult" the encoder's understanding of the input at every step of generation. Each decoder block contains:

1. Layer normalization
2. Masked self-attention (causal, so tokens only attend to previous positions)
3. Residual connection
4. Layer normalization
5. Cross-attention to encoder outputs
6. Residual connection
7. Layer normalization
8. Position-wise feed-forward network
9. Residual connection

The masking in self-attention ensures the decoder can only see tokens it has already generated, maintaining the autoregressive property needed for text generation. Without this mask, the model could "cheat" during training by looking at future tokens, learning to copy rather than predict. Cross-attention allows each decoder position to attend to all encoder positions, integrating the input representation into the generation process. When generating a German translation, each German word can attend to all the English words, determining which parts of the source are most relevant for creating the current output token.

The interaction between masked self-attention and cross-attention is worth examining carefully. The self-attention sublayer lets the decoder build coherent internal context: "given what I have generated so far, what is the current state of my output?" The cross-attention sublayer then anchors that internal context to the source: "given my current generation state, which parts of the input should inform my next word?" These two sources of information, the decoder's own history and the encoder's representation of the source, combine in the feed-forward layers to produce the final output distribution. This two-source structure is what makes encoder-decoder models particularly well-suited for tasks with a clear input-output separation, as opposed to open-ended generation where there is no distinct "source" to attend to.

The complete forward pass proceeds through a two-stage process that first builds understanding and then produces output. First, input tokens are embedded and passed through all encoder layers. Each encoder layer refines the representations, with early layers capturing local syntactic patterns and deeper layers building more abstract semantic representations. The encoder's final hidden states, one vector per input token, become the "memory" that the decoder will reference throughout generation.

During generation, the decoder receives the previously generated tokens (or just a start token initially). These pass through self-attention layers with causal masking, letting the model to consider what it has already said while deciding what to say next. Then cross-attention layers query the encoder memory, determining which parts of the input are relevant for generating the current token. The final decoder hidden state projects to vocabulary logits, and the highest-probability token becomes the next output. This process repeats, with each new token extending the decoder's self-attention context, until the model produces a stop token or reaches a maximum length.

The key insight is that encoder computation and decoder computation are interleaved at training time but separated at inference time. During training, the encoder runs once per example, and the decoder runs in parallel over all target positions (using teacher forcing, where the ground truth target is fed in rather than the model's own predictions). During inference, the encoder runs once, but the decoder must run autoregressively, one step at a time, because each new token depends on all previous outputs. This asymmetry means that inference speed is dominated by decoder steps, and techniques for accelerating autoregressive generation (like key-value caching) are necessary for deploying T5 in production settings.