Prefix Language Modeling › Encoder-Decoder vs Decoder-Only Prefix LM

Michael BrenndoerferUpdated July 14, 20252 min read
Language AI HandbookAll chapters
Part 22 of 67Pre-training Objectives
Chapter 159Prefix Language Modeling › Encoder-Decoder vs Decoder-Only Prefix LM
Reading section 10 of 16 · Encoder-Decoder vs Decoder-Only Prefix LM · 2 minSection 10 of 16 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 159 of 513

Section 10 of 16 · 2 min read

Prefix Language Modeling › Encoder-Decoder vs Decoder-Only Prefix LMOne section · Sections in this chapter10 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Prefix LM can be implemented in two architectural styles, each with distinct trade-offs.

Encoder-decoder models like T5 process the prefix through a bidirectional encoder, then pass encoded representations to a causal decoder that generates the target. The encoder and decoder are separate modules with cross-attention connecting them.

Decoder-only models like GPT use a single transformer stack with the prefix LM attention mask. The same weights process both prefix and target, with the attention mask controlling information flow.

Out[26]:
Visualization
Two diagrams showing encoder-decoder and decoder-only architectures for prefix language modeling.
Two architectural approaches to prefix LM. Encoder-decoder (left) uses separate modules with cross-attention, while decoder-only (right) uses a single transformer with a hybrid attention mask. The encoder-decoder approach allows more specialization, while decoder-only is simpler and often more parameter-efficient.

The choice between architectures depends on the application. Encoder-decoder models can process very long prefixes efficiently since encoder states are computed once and reused. Decoder-only models are simpler to implement and often perform comparably for shorter contexts.