Prefix Language Modeling › The Prefix LM Intuition
Language AI HandbookAll chapters Prefix Language Modeling › The Prefix LM IntuitionOne section · Sections in this chapter2 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Consider a translation task: "Translate to French: The cat sat on the mat." The input prompt is complete and known. When generating "Le chat s'est assis sur le tapis," each French word depends on the entire English prompt plus previously generated French words. There's no benefit to hiding "mat" from "cat" in the English prompt.
Prefix LM captures this asymmetry. The sequence is split into two parts:
- Prefix: The conditioning context (prompt, input, instruction). Tokens here can attend to each other bidirectionally.
- Target: The continuation to be generated. Tokens here can attend to the full prefix plus their own left context, following causal constraints.
This hybrid attention pattern preserves the generation capability of causal LM while allowing richer representation of the prefix through bidirectional attention.
This structure matches many real-world scenarios. In question answering, the question is the prefix and the answer is the target. In summarization, the document is the prefix and the summary is the target. In dialogue, the conversation history is the prefix and the next response is the target. The bidirectional prefix captures the full context, while the causal target enables coherent generation.