Causal Language Modeling › Summary
Michael BrenndoerferUpdated July 7, 20251 min read
Language AI HandbookAll chapters Part 22 of 67Pre-training Objectives
Causal Language Modeling › SummaryOne section · Sections in this chapter12 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Causal language modeling trains models to predict the next token given all previous tokens. This chapter covered the key concepts:
- Autoregressive factorization decomposes sequence probability into a product of conditional probabilities, each predicting one token from its left context
- The CLM objective minimizes cross-entropy loss between predicted and actual next tokens. This gives dense gradient signals from every position
- Causal masking in attention layers enforces the left-to-right information flow, preventing the model from seeing future tokens during training
- Training data quality and scale directly determine model capabilities, with modern LLMs consuming trillions of curated tokens
- Scaling laws show predictable improvements as compute and data, along with parameters, increase, following power-law relationships
- Perplexity measures model quality as the exponential of average loss, with lower values indicating better predictions
The next chapter explores masked language modeling, a bidirectional alternative that trades generation capability for richer contextual representations.