Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Pre-Norm vs Post-Norm › Hybrid Approaches

Michael BrenndoerferUpdated June 10, 20252 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 106Pre-Norm vs Post-Norm › Hybrid Approaches

Reading section 13 of 18 · Hybrid Approaches · 2 minSection 13 of 18 · 2 min

Section 13 of 18 · 2 min read

Pre-Norm vs Post-Norm › Hybrid ApproachesOne section · Sections in this chapter13 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Recent work has explored combining the benefits of both approaches. Researchers have proposed several hybrids, motivated by the desire to get pre-norm's stability alongside post-norm's normalized output guarantees.

The scaled residual approach reduces the magnitude of each sublayer's contribution before the residual addition. The idea is that large sublayer outputs are the main source of post-norm instability: if each sublayer only makes a small perturbation to the residual stream, the LayerNorm Jacobian operates on a more stable signal. The cost is slower learning in the early training phase.

One variant uses post-norm with scaled residuals:

In[26]:

Code

def scaled_post_norm_block(x, sublayer_fn, gamma, beta, alpha=0.1):
    """
    Scaled post-norm: reduces residual contribution for stability.

    Args:
        alpha: Scaling factor for sublayer output (typically 0.1-0.5)
    """
    sublayer_output = sublayer_fn(x)
    residual_sum = x + alpha * sublayer_output
    output = layer_norm(residual_sum, gamma, beta)
    return output

Another approach, used in some recent models, applies normalization both before and after the sublayer:

In[27]:

Code

def sandwich_norm_block(x, sublayer_fn, gamma1, beta1, gamma2, beta2):
    """
    Sandwich normalization: LayerNorm before and after sublayer.
    Provides extra stability at the cost of additional computation.
    """
    # Pre-normalize
    x_norm = layer_norm(x, gamma1, beta1)

    # Apply sublayer
    sublayer_output = sublayer_fn(x_norm)

    # Post-normalize the sublayer output
    sublayer_output = layer_norm(sublayer_output, gamma2, beta2)

    # Add residual
    output = x + sublayer_output

    return output

These hybrid approaches add complexity but can provide benefits in specific situations, particularly for extremely deep or wide networks. The sandwich norm approach has shown promise in some experimental architectures because it combines the clean inputs to the sublayer (from the pre-norm step) with the clean outputs from the sublayer (from the post-norm step), while still passing the residual through unnormalized. This gives maximum control over what the sublayer sees and produces, at the cost of two normalization operations per block. Whether the added computational cost is worth the stability benefit depends on the specific training regime and model scale.

In production systems, most practitioners default to pre-norm and tune it before considering a hybrid. Hybrids introduce additional hyperparameters, such as the alpha scaling factor, which require their own tuning. The general principle is to prefer simpler architectures unless you have a specific, well-characterized problem that a more complex design solves.