Pre-Norm vs Post-Norm › Visualizing the Difference

Michael BrenndoerferUpdated June 10, 20253 min read
Language AI HandbookAll chapters
Part 15 of 67Transformer Blocks
Chapter 106Pre-Norm vs Post-Norm › Visualizing the Difference
Reading section 4 of 18 · Visualizing the Difference · 3 minSection 4 of 18 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 106 of 513

Section 4 of 18 · 3 min read

Pre-Norm vs Post-Norm › Visualizing the DifferenceOne section · Sections in this chapter4 / 18 ↓
Sections in this chapter18
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The structural difference between pre-norm and post-norm becomes clearer when we visualize the computation graphs:

Out[9]:
Visualization
Block diagram showing post-norm flow: input to sublayer to add to layer norm to output.
Post-norm: The input flows through the sublayer, combines with the residual, then gets normalized. Layer normalization is the last operation.
Block diagram showing pre-norm flow: input to layer norm to sublayer to add with original input to output.
Pre-norm: The input is normalized before the sublayer. The residual connection adds the original input, bypassing normalization entirely.

The diagrams reveal the fundamental difference. In post-norm, the residual enters the add block alongside the sublayer output, and layer normalization processes their sum. In pre-norm, the residual bypasses both layer normalization and the sublayer, creating a direct path from input to output.