Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Decoder Architecture › Decoder Architecture Comparison

Michael BrenndoerferUpdated June 19, 20252 min read

Language AI HandbookAll chapters

Part 16 of 67Transformer Architectures

Chapter 112Decoder Architecture › Decoder Architecture Comparison

Reading section 11 of 16 · Decoder Architecture Comparison · 2 minSection 11 of 16 · 2 min

Section 11 of 16 · 2 min read

Decoder Architecture › Decoder Architecture ComparisonOne section · Sections in this chapter11 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Different models use slightly different decoder designs. The core pattern is consistent, but details vary:

Decoder architecture comparison across popular language models. All models share the same fundamental structure but differ in scale and specific design choices.
Model	Layers	Heads	$d_{\text{model}}$	FFN Ratio	Norm	Notable Features
GPT-2 Small	12	12	768	4x	Layer Norm	Learned position embeddings
GPT-2 Medium	24	16	1024	4x	Layer Norm	Pre-LN architecture
GPT-3	96	96	12288	4x	Layer Norm	Sparse attention in some heads
Llama 2 7B	32	32	4096	2.7x	RMSNorm	RoPE, SwiGLU FFN
Llama 3 8B	32	32	4096	3.5x	RMSNorm	Grouped-query attention
Mistral 7B	32	32	4096	3.5x	RMSNorm	Sliding window attention

Modern decoders favor RMSNorm over LayerNorm for efficiency, SwiGLU or GeGLU activations in the FFN for better performance, and positional encodings like RoPE that generalize to longer sequences. Grouped-query attention reduces the KV cache size by sharing key-value projections across groups of heads.