Transformer Block Assembly › The Standard Transformer Block

A transformer block is a modular unit that can be stacked to build arbitrarily deep models. Each block takes a sequence of token representations as input and produces a refined sequence as output. The refinement happens through two complementary operations: attention (which mixes information across positions) and the feed-forward network (which transforms each position independently).

The modern pre-norm block, used in GPT and LLaMA as well as most contemporary language models, follows a specific structure. The computation proceeds in two stages (first attention, then feed-forward), with each stage wrapped in a residual connection:

where:

• $\mathbf{x} \in \mathbb{R}^{n \times d}$: the input to the block, a matrix containing $n$ tokens (one per row) with $d$-dimensional representations
• $\text{Norm}(\cdot)$: normalization function, typically RMSNorm in modern architectures, which stabilizes activations before each sublayer
• $\text{Attention}(\cdot)$: multi-head self-attention, which allows each token to gather information from other positions in the sequence
• $\text{FFN}(\cdot)$: the feed-forward network, which transforms each token independently through nonlinear layers (potentially with gating like SwiGLU)
• $\mathbf{h} \in \mathbb{R}^{n \times d}$: intermediate representation after the attention sublayer, combining the original input with attention's contribution
• $\mathbf{y} \in \mathbb{R}^{n \times d}$: the block output, passed to the next block in the stack

The structure of each sublayer follows the same pattern: (1) normalize the input, (2) apply the transformation (attention or FFN), and (3) add the result back to the original input via a residual connection. This "normalize → transform → add" pattern repeats for both sublayers.

Placing normalization before each sublayer (pre-norm) lets gradients flow directly through the residual path without being distorted by normalization. This ordering, which we analyzed in the pre-norm vs post-norm chapter, is important for training deep transformer models (50+ layers).