Causal Language Modeling › Causal Language Modeling

Language models learn to predict the next word. This simple objective, applied at massive scale, has produced the most capable AI systems ever built. GPT-4 and Claude, along with LLaMA, share this foundation with virtually every modern generative model: given a sequence of tokens, predict what comes next.

Causal Language Modeling (CLM) is the formal name for this training objective. The "causal" refers to the direction of information flow: predictions depend only on past tokens, never on future ones. This constraint mirrors how humans produce language, word by word, and makes the learned model directly usable for text generation without any modifications to the architecture at inference time.

CLM derives its training signal directly from raw text. You don't need labeled data, annotated categories, or hand-crafted features. Every piece of raw text is automatically a supervised learning example: the input is a prefix, and the label is the next token. This means the entire internet, all books ever digitized, every line of code ever written can be converted into training signal at zero additional annotation cost. The web itself becomes a teacher.

Think of it this way: every time a model reads the phrase "The Eiffel Tower is located in," it is tested on whether it predicts "Paris" as the next token. Every time it reads import numpy as np\nimport, it is tested on whether it knows common Python libraries. Every sentence in every document becomes a series of prediction problems, and the model that learns to solve all of them must, necessarily, have learned a great deal about how language works.

The simplicity of the objective also means it generalizes remarkably. A model that learns to predict text must represent grammar, factual knowledge, reasoning patterns, and even emotional tone, because all of these factors influence what word comes next. CLM is in this sense a universal objective: it forces the model to internalize whatever statistical regularities are needed to predict the next token accurately, across all contexts and all domains present in the training corpus.

In this chapter, we'll unpack the mathematics behind CLM, understand why it works so well, implement the loss function from scratch, and explore the training data and scaling properties that have driven recent breakthroughs.