Causal Language Modeling: The Foundation of Generative AI

Language models learn to predict the next word. This simple objective, applied at massive scale, has produced the most capable AI systems ever built. GPT-4 and Claude, along with LLaMA, share this foundation with virtually every modern generative model: given a sequence of tokens, predict what comes next.

Causal Language Modeling (CLM) is the formal name for this training objective. The "causal" refers to the direction of information flow: predictions depend only on past tokens, never on future ones. This constraint mirrors how humans produce language, word by word, and makes the learned model directly usable for text generation without any modifications to the architecture at inference time.

CLM derives its training signal directly from raw text. You don't need labeled data, annotated categories, or hand-crafted features. Every piece of raw text is automatically a supervised learning example: the input is a prefix, and the label is the next token. This means the entire internet, all books ever digitized, every line of code ever written can be converted into training signal at zero additional annotation cost. The web itself becomes a teacher.

Think of it this way: every time a model reads the phrase "The Eiffel Tower is located in," it is tested on whether it predicts "Paris" as the next token. Every time it reads import numpy as np\nimport, it is tested on whether it knows common Python libraries. Every sentence in every document becomes a series of prediction problems, and the model that learns to solve all of them must, necessarily, have learned a great deal about how language works.

The simplicity of the objective also means it generalizes remarkably. A model that learns to predict text must represent grammar, factual knowledge, reasoning patterns, and even emotional tone, because all of these factors influence what word comes next. CLM is in this sense a universal objective: it forces the model to internalize whatever statistical regularities are needed to predict the next token accurately, across all contexts and all domains present in the training corpus.

In this chapter, we'll unpack the mathematics behind CLM, understand why it works so well, implement the loss function from scratch, and explore the training data and scaling properties that have driven recent breakthroughs.

Causal language modeling is built on a mathematical framework called the autoregressive factorization. Before diving into the training objective or the loss function, it helps to understand this underlying structure, because it explains both why the approach is tractable and why it naturally produces generative models. The central challenge is assigning probabilities to sequences of arbitrary length. The autoregressive factorization is the field's mathematically exact solution rather than an approximation.

How do you assign a probability to an entire sentence? This is the fundamental question that language models must answer. Given a sequence of tokens $x_1, x_2, \ldots, x_n$, we want to compute $P(x_1, x_2, \ldots, x_n)$, the probability that this particular sequence occurs in natural language.

Consider what this means in practice. With a vocabulary of 50,000 tokens and sequences of length 100, we'd need to assign probabilities to $50000^{100}$ possible sequences. That's more possibilities than atoms in the observable universe, raised to a power larger than the universe's age in seconds. Storing or computing such a distribution directly is impossible.

We need a way to break this intractable joint probability into manageable pieces. Fortunately, probability theory gives us exactly such a tool: the chain rule.

The chain rule states that any joint probability can be decomposed into a product of conditional probabilities. For a sequence, this means:

Let's trace through what each term represents:

1. $P(x_1)$: The probability of the first token appearing at the start of a sequence. This has no conditioning context.
2. $P(x_2 | x_1)$: Given we've seen the first token, what's the probability of the second?
3. $P(x_3 | x_1, x_2)$: Given the first two tokens, what comes third?
4. And so on, until $P(x_n | x_1, \ldots, x_{n-1})$: the final token, conditioned on everything before it.

This telescoping product captures the sequential nature of language. Each new word depends on what came before, exactly matching our intuition about how text is generated.

We can express this factorization more concisely using product notation:

where:

• $x_t$: the token at position $t$ in the sequence
• $x_{<t}$: all tokens before position $t$, that is, the sequence $(x_1, x_2, \ldots, x_{t-1})$
• $P(x_t | x_{<t})$: the conditional probability of token $x_t$ given all preceding tokens

For the first position where $t=1$, we define $x_{<1}$ as empty, so $P(x_1 | x_{<1}) = P(x_1)$.

Each factor $P(x_t | x_{<t})$ is a conditional distribution over the entire vocabulary. Given everything we've seen so far, what's the probability of each possible next token? This is precisely the question a language model learns to answer.

This decomposition is mathematically exact, not an approximation. The chain rule holds for any joint distribution. What makes it practical is that we've transformed an impossible problem (representing $50000^{100}$ probabilities) into a tractable one (learning a function that outputs a distribution over 50,000 tokens given any context).

The key insight is that while contexts vary enormously, they share statistical patterns. The word "the" tends to be followed by nouns. Questions end with question marks. Technical documents use technical vocabulary. A neural network can learn these patterns and generalize them to new contexts it has never seen before.

There is also a deep connection between this factorization and causality, which is where the name "causal" language modeling comes from. When a model predicts $P(x_t | x_{<t})$, it is computing what should come next based solely on the causal history, everything that appeared before. No information from the future leaks into the prediction. This mirrors real-world language generation: when a human writes a sentence, the word they choose at each moment is influenced by what they have already written, not by what they have not yet written. CLM formalizes this intuitive notion and enforces it architecturally through causal masking, which we'll cover later in the chapter.

Notice that the factorization also implies a natural sampling procedure. To generate a sequence, you sample $x_1$ from $P(x_1)$, then sample $x_2$ from $P(x_2 | x_1)$, then $x_3$ from $P(x_3 | x_1, x_2)$, and so on. At each step you need only the distribution over the next single token, which the model can compute efficiently. This is why the same model used for training, which minimizes prediction loss, can be used directly for generation without any modifications.

Now that we understand how to decompose sequence probability, we need a way to train a model to produce good probability estimates. This requires two things: a loss function that measures prediction quality, and a mechanism to improve the model based on that measurement. The CLM objective is the standard answer to both. It is derived from maximum likelihood estimation, a classical statistical principle, applied to the autoregressive factorization we just derived. Understanding the derivation step by step reveals why the final formula looks the way it does and why the choices made along the way are the right ones.

Our goal is to find model parameters $\theta$ that make the training data as probable as possible. Given a training sequence $x_1, \ldots, x_n$, we want to maximize:

Maximizing a product of many small probabilities is numerically unstable. As sequences grow longer, the product shrinks toward zero, causing underflow. The standard solution is to work with logarithms. Since $\log$ is monotonically increasing, maximizing a probability is equivalent to maximizing its log:

The product becomes a sum, which is numerically stable and computationally convenient. Now, optimization algorithms typically minimize rather than maximize, so we flip the sign to get our loss function:

where:

• $\mathcal{L}(\theta)$: the loss function we minimize during training
• $\theta$: the model parameters (weights and biases of the neural network)
• $n$: the length of the training sequence
• $P_\theta(x_t | x_{<t})$: the probability the model assigns to the correct token $x_t$ given the context $x_{<t}$
• $\log P_\theta(x_t | x_{<t})$: the log-probability, which is negative since probabilities lie in $(0, 1)$

This loss function has a beautiful interpretation: it's the cross-entropy between the model's predicted distribution and the true distribution (which puts all probability mass on the actual next token).

To see why, recall that cross-entropy measures how well a predicted distribution $Q$ matches a true distribution $P$:

At each position $t$, the "true distribution" is a one-hot vector: probability 1 for the actual token $x_t$, and 0 for everything else. The model predicts a distribution $P_\theta(\cdot | x_{<t})$ over all vocabulary tokens. The cross-entropy simplifies to:

Only the true token's probability matters. This is why cross-entropy loss is also called "negative log-likelihood" in language modeling contexts.

The loss function creates an intuitive learning signal. When the model assigns high probability to the correct token, the loss contribution is small. When the model is surprised, the loss contribution is large.

Consider these scenarios:

• Confident and correct ($P = 0.9$): $-\log(0.9) \approx 0.1$. Small loss, weak gradient. The model is doing well here.
• Uncertain ($P = 0.1$): $-\log(0.1) \approx 2.3$. Moderate loss. Room for improvement.
• Confident and wrong ($P = 0.01$): $-\log(0.01) \approx 4.6$. Large loss, strong gradient. The model needs to update significantly.

This asymmetry is powerful: the model receives the strongest teaching signal precisely where it's making the biggest mistakes.

A remarkable property of CLM loss is that it decomposes across positions. A single sequence of length $n$ provides $n$ independent gradient signals, one for each prediction. This is dramatically more efficient than classification tasks where a single input yields a single label.

Consider training on a document with 1,000 tokens. Each forward pass produces 1,000 predictions and 1,000 loss terms. The model learns from every token simultaneously, extracting maximum information from the training data. This efficiency is one reason language models can learn so much from their training corpora.

Let's implement this loss function to see how it works in practice:

A key insight of CLM is that a single sequence yields multiple training examples. For a sequence of length $n$, we predict tokens at positions 2 through $n$ using contexts of increasing length. This is what makes CLM so data-efficient compared to other supervised learning tasks. In image classification, a single image is a single training example. In CLM, a single sentence of 20 words provides 19 training examples, and a document of 2,000 tokens provides 1,999. The same bytes of raw text generate orders of magnitude more gradient signal than would be possible with explicit labeling.

Consider the sentence "The cat sat on the mat". We create training pairs:

The model sees the same sequence but learns from every position simultaneously. This is implemented using a clever shifting trick:

The inputs and targets are offset by one position. At position $i$, the model receives tokens $0, 1, \ldots, i$ as input and predicts token $i+1$. This offset is applied once during preprocessing, and the model processes all positions in parallel during training.

In practice, this parallel processing is why transformers are so much faster to train than recurrent neural networks. An RNN would process the sequence step by step, computing each hidden state sequentially before moving to the next. A transformer with causal masking computes all hidden states in a single forward pass, using the mask to enforce the left-to-right constraint. This enables much larger batch sizes and dramatically higher GPU utilization.

For the autoregressive factorization to hold, the model must not "peek" at future tokens when predicting the current one. In transformer architectures, this is enforced through causal masking in the attention mechanism. Without this constraint, a model could trivially solve the prediction problem by looking ahead at the answer, which would produce a model that learns nothing useful and cannot generate text at test time.

The challenge is a structural one. Standard transformer attention is fully bidirectional: every position attends to every other position. This is powerful for understanding tasks, where you want to incorporate all available context. But for generation, it creates a problem. If position 3 can see positions 4, 5, and 6 during training, the model learns to use that future information. At inference time, positions 4, 5, and 6 have not been generated yet, so the model encounters a distribution shift it was never trained to handle.

Causal masking solves this by zeroing out attention from any position to any future position. The model is structurally prevented from accessing information it would not have access to during generation. In practice, this is implemented by adding a large negative constant (negative infinity) to the attention scores for disallowed positions before applying the softmax. After softmax, those positions receive attention weight of zero, as if they do not exist.

The attention pattern looks like this:

The negative infinity values become zero probability after softmax, effectively blocking information flow from future positions. Each row sums to 1.0, distributing attention only over valid (past and present) positions.

Let's train a minimal causal language model to see the complete pipeline. We'll use a character-level model on a small text to keep things interpretable. A character-level model treats each individual character as a token rather than each word or subword. This gives us a vocabulary of only 26 letters plus punctuation, making the prediction problem simple enough to solve with a tiny model in a few hundred steps. The core CLM machinery: the autoregressive factorization, the cross-entropy loss, the causal masking, and the generation loop, is identical to what large-scale models use. Only the vocabulary and capacity differ.

The generated text isn't perfect, but the model has learned character-level patterns from just 170 characters of Shakespeare. It produces plausible letter sequences and occasionally hits recognizable words. With more data and capacity, this same objective scales to GPT-4.

The quality and scale of training data fundamentally shapes what a causal language model learns. Modern LLMs are trained on datasets containing trillions of tokens, carefully curated from diverse sources. Understanding where that data comes from, how it is processed, and what tradeoffs different sources introduce gives you a much clearer picture of why different models behave the way they do.

Common training data sources include:

• Web crawls: Common Crawl and C4, plus similar filtered web scrapes, provide broad coverage of internet text. Heavy filtering removes spam and duplicates, along with low-quality content.
• Books: Project Gutenberg, Books3, and licensed book corpora provide long-form, well-edited text that teaches narrative structure and coherent reasoning.
• Code: GitHub, Stack Overflow, and code documentation help models understand programming languages and technical reasoning.
• Scientific literature: Papers from arXiv and PubMed, as well as Semantic Scholar, provide technical depth and formal reasoning.
• Curated datasets: Wikipedia, news articles, and human-written examples balance quality with scale.

Data quality matters enormously. A model trained on Reddit comments writes like Reddit. A model trained on textbooks writes like textbooks. The mixture of sources directly influences the model's capabilities and style, as well as its failure modes. This is why organizations release detailed data cards alongside their models, and why replicating a model's behavior often requires replicating its data mixture.

Deduplication is critical: repeated text causes models to memorize rather than generalize. Modern pipelines use MinHash, exact substring matching, or embedding-based deduplication to remove near-duplicate documents. Without deduplication, a model may see certain passages thousands of times during training, learning to recite them verbatim rather than learning the underlying patterns. The Pile and RedPajama datasets, for example, both describe extensive deduplication pipelines as a first-class concern, not an afterthought.

Raw text cannot be fed directly to a language model. The first step is tokenization: converting text into a sequence of integer token IDs drawn from a fixed vocabulary. Modern models use subword tokenization schemes like Byte Pair Encoding (BPE) or SentencePiece. These methods split rare words into smaller pieces while keeping common words intact. The word "tokenization" might become "token", "ization" as two subword units, while "the" remains a single token. This balances vocabulary size against coverage: a vocabulary of around 50,000 to 100,000 tokens is standard, large enough to represent most content efficiently but small enough for the output projection layer to remain manageable.

After tokenization, documents are typically concatenated into long streams and then chunked into fixed-length windows, often 2,048 or 4,096 tokens. This ensures the model sees context that spans document boundaries, which helps it learn transition patterns and avoids the artificial constraint of always resetting context at document edges. The resulting data loader simply iterates through these fixed-length windows during training, which is highly efficient for GPU throughput.

Not all data is created equal for training. Raw web data contains enormous amounts of low-value content: spam, boilerplate HTML, generated text, near-duplicate articles. High-quality sources like curated books and peer-reviewed papers improve reasoning and factual accuracy but are available in smaller quantities. Training on only high-quality data produces models that write elegantly but may lack breadth. Training on only web data produces models with broad knowledge but inconsistent quality.

The practical answer is careful mixture weighting. Models like LLaMA 3 weight code-heavy data higher than raw web crawl because code trains logical and structured reasoning that transfers broadly. Wikipedia and books typically receive upsampling, meaning the model sees them multiple times per training run despite being a small fraction of the raw data volume. Getting this mixture right is an active area of research, and small changes in data weights can produce measurable differences in downstream benchmark performance.

Perhaps the most remarkable property of CLM is how predictably it scales. As we increase model size, dataset size, and compute, performance improves following consistent power laws. This predictability is what distinguishes language model scaling from most engineering disciplines. Usually, when you make something bigger, you encounter bottlenecks, diminishing returns, and phase transitions that are difficult to predict. CLM scaling instead follows remarkably smooth trajectories, making it possible to predict the behavior of a trillion-parameter model by fitting a curve to experiments with much smaller models.

The scaling laws discovered by Kaplan et al. (2020) and refined by Hoffmann et al. (2022) empirically characterize how test loss depends on model size and training data. The key finding is that loss follows a power-law relationship with both factors:

where:

• $L(N, D)$: the cross-entropy loss on held-out test data, as a function of model size and data
• $N$: the number of trainable model parameters (excluding embeddings)
• $D$: the number of training tokens the model has seen
• $\alpha_N \approx 0.076$: the scaling exponent for model size (how quickly loss improves as $N$ grows)
• $\alpha_D \approx 0.095$: the scaling exponent for data (how quickly loss improves as $D$ grows)
• $N_c$ and $D_c$: fitted constants that set the scale (roughly $8.8 \times 10^{13}$ and $5.4 \times 10^{13}$ respectively)
• $L_\infty$: the irreducible loss, representing fundamental uncertainty in language that no model can eliminate

The formula has three additive terms. The first term $\left(\frac{N_c}{N}\right)^{\alpha_N}$ captures model capacity limitations: smaller models have higher loss. The second term $\left(\frac{D_c}{D}\right)^{\alpha_D}$ captures data limitations: less training data means higher loss. The third term $L_\infty$ is the floor, around 1.69 nats, representing the inherent unpredictability of natural language.

This equation reveals that loss decreases as a power law with both model size and data. There's no plateau in sight: 10x more compute yields roughly 0.1 lower loss, consistently across many orders of magnitude.

The practical implication is clear: if you want a better language model, train a bigger model on more data with more compute. This insight has driven the race to scale, from GPT-2's 1.5 billion parameters to models with hundreds of billions.

The original Kaplan et al. scaling laws suggested that given a fixed compute budget, you should spend most of it on model size and train on relatively little data. GPT-3, trained with 175 billion parameters on roughly 300 billion tokens, reflected this philosophy.

Hoffmann et al. (2022), in a paper introducing the Chinchilla model, revised this picture significantly. Their experiments showed that the optimal training strategy is to scale model size and data size proportionally. For every doubling of parameters, you should also double the number of training tokens. By this analysis, GPT-3 was substantially undertrained: a much smaller model trained on far more data would have achieved equal performance at much lower cost.

This finding reoriented the field. Subsequent models like LLaMA and Mistral, along with Gemma, were designed with the Chinchilla recipe in mind, training smaller models on larger datasets. A 7-billion-parameter model trained on 2 trillion tokens can outperform a 175-billion-parameter model trained on 300 billion tokens at a fraction of the inference cost. The same compute budget can produce a much more efficient model if allocated correctly between parameters and data.

In practice, the Chinchilla optimal point assumes you train the model once and then stop. But inference matters too. If a model will be deployed and run a billion times, a smaller model is much more valuable than a larger one with equal loss, because every inference query is cheaper. This "inference-optimal" perspective has pushed the field even further toward smaller, data-rich models.

Scaling also improves emergent capabilities. Models below a certain size cannot perform multi-step reasoning, follow complex instructions, or write working code. Above threshold scales, these abilities appear suddenly, a phenomenon called emergence. Whether emergence is truly discontinuous or an artifact of how we measure it remains debated, but the practical reality is clear: beyond certain parameter and data thresholds, qualitatively new behaviors appear that could not be reliably predicted from smaller-scale experiments.

Perplexity is the standard evaluation metric for language models. While cross-entropy loss is useful for training, perplexity provides a more interpretable measure of model quality. It answers the question: on average, how many tokens is the model choosing between at each step? A perfect model with perplexity 1.0 would always predict the next token with probability 1.0, never being surprised. A completely random model over a 50,000-token vocabulary would have perplexity 50,000. Real models sit somewhere in between, with state-of-the-art models achieving single-digit perplexities on well-studied benchmarks.

Perplexity is defined as the exponential of the average negative log-likelihood:

where:

• $\text{PPL}$: perplexity, the evaluation metric (lower is better)
• $N$: the total number of tokens in the evaluation dataset
• $P(x_i | x_{<i})$: the probability the model assigns to token $x_i$ given its context
• $-\frac{1}{N}\sum_{i=1}^{N} \log P(x_i | x_{<i})$: the average cross-entropy loss per token

The exponential converts the average log-probability back to a probability-like scale. If the model achieves an average loss of 2.3 nats per token, the perplexity is $e^{2.3} \approx 10$.

The key intuition is this: a perplexity of 10 means the model is, on average, as uncertain as if it were choosing uniformly among 10 equally likely options at each step. A perplexity of 100 would mean 100-way uncertainty. State-of-the-art models achieve perplexities below 10 on standard benchmarks like WikiText-103, meaning they often predict the correct next word with high confidence.

This perplexity on training data shows how well the model has fit the corpus. Since we're evaluating on the same text we trained on, this is an optimistic estimate. Held-out test perplexity would be higher. This reflects true generalization ability.

One important subtlety is that perplexity comparisons across models are only valid when the tokenizers are identical. A model with a larger vocabulary tends to tokenize text into fewer tokens, each carrying more information, and will therefore show different perplexity numbers even if the underlying prediction quality is the same. A model with 100,000-token vocabulary might achieve perplexity 5 on the same text where a model with 32,000-token vocabulary achieves perplexity 8, simply because the larger vocabulary tokenizes less aggressively. When comparing language models from different organizations, always check that the evaluation setup controls for tokenization differences.

Causal language modeling has revolutionized NLP, but it comes with important limitations that shape how we use these models in practice. Understanding these limitations is not just academic. They determine which applications work well, which require engineering workarounds, and which are fundamentally unsuited to CLM-based systems.

The unidirectional constraint means CLM models cannot naturally incorporate future context. For tasks like filling in the middle of a sentence or bidirectional understanding, this is a fundamental limitation. Models like BERT use masked language modeling to capture bidirectional dependencies, trading generation capability for richer representations. In practice, many applications now use CLM models with careful prompting to work around this constraint. However, the constraint also creates an important advantage: CLM models can generate text of arbitrary length without architectural changes, because each new token only requires attending to the growing context window behind it.

Training on next-token prediction creates models that are excellent at mimicking patterns in training data but may struggle with factual accuracy. A model can fluently generate text about events that never happened or facts that are simply wrong. The objective optimizes for plausibility, not truth. This has led to significant research in retrieval augmentation and grounding techniques that anchor model outputs in verified information. The core problem is that a confident, fluent sentence and a confident, fluent lie are indistinguishable from the perspective of the CLM loss function. Both are rewarded equally if the tokens flow naturally from the preceding context.

The context length limitation is a real practical constraint. Standard transformer attention grows quadratically with sequence length, making very long contexts expensive. A sequence of 4,096 tokens requires roughly 16 times as much memory for attention than a sequence of 1,024 tokens. This means early CLM models could not maintain coherent reasoning over very long documents. Modern research on efficient attention (sparse attention, sliding window attention, linear attention variants) aims to extend this horizon, and recent models like Gemini 1.5 have pushed context lengths to millions of tokens, though at significant engineering cost.

The compute requirements are staggering. Training frontier models costs tens of millions of dollars and consumes megawatt-hours of electricity. This concentrates capability in a few well-resourced organizations and raises sustainability concerns. Techniques like distillation and quantization, along with efficient architectures, aim to democratize access, but the gap between frontier and accessible models remains wide. Training GPT-3 was estimated to cost over 4 million dollars in cloud compute in 2020, and frontier models since then have cost substantially more.

Despite these limitations, CLM has enabled capabilities that seemed impossible a decade ago. Modern LLMs can write code, translate languages, answer questions, and engage in open-ended conversation. They serve as foundations for instruction-following and reasoning, as well as tool-using agents. The simplicity of the objective belies the complexity of what emerges from optimizing it at scale. The same trained model supports both zero-shot prompting (where you simply ask the model to do something) and few-shot learning (where you demonstrate the task with examples), making it flexible across downstream applications.

When training causal language models, a number of parameters significantly impact performance. Some are architectural, fixed at model design time. Others are optimization hyperparameters that you tune during training. Understanding what each controls helps you reason about tradeoffs without needing to memorize specific values.

The architectural parameters control model capacity:

• d_model: The hidden dimension of the transformer. Larger values (512, 768, 1024) increase capacity but require more compute. Our tiny model used 64. The hidden dimension determines the richness of the internal representations: a larger d_model gives the model more "space" to encode different aspects of meaning.
• n_heads: Number of attention heads. Should divide d_model evenly. More heads allow the model to attend to different aspects of context simultaneously. Each head learns a different pattern: one might track syntactic structure, another co-reference, another topical continuity.
• n_layers: Depth of the transformer stack. Deeper models can learn more complex patterns but are slower to train. Production models use 12-96 layers. Early layers tend to capture local syntactic patterns, while later layers capture more abstract semantic relationships.

The training hyperparameters control optimization:

• learning_rate: Typically 1e-4 to 6e-4 for transformers. Higher rates speed training but risk instability. Warmup schedules help stabilize early training by gradually increasing the learning rate from near zero over the first few thousand steps, allowing the model to settle into a stable optimization trajectory before taking large gradient steps.
• batch_size: Larger batches provide more stable gradients but require more memory. Modern LLMs use effective batch sizes in the millions of tokens, often achieved through gradient accumulation across many smaller physical batches when GPU memory is insufficient to hold a large batch directly.
• seq_len (context length): Maximum sequence length the model can process. Longer contexts enable better understanding but increase memory quadratically with attention. For a model with context length 4,096, each forward pass processes up to 4,096 tokens, but attention requires storing a 4,096 by 4,096 matrix per layer.

The generation-time parameter controls sampling:

• temperature: Controls randomness during generation. Values near 0 produce deterministic, repetitive output. Values near 1 produce diverse, sometimes incoherent text. Typical range: 0.7-1.0. The temperature divides the logits before the softmax: lower temperature sharpens the distribution toward the top prediction, while higher temperature flattens it, giving unlikely tokens a greater chance of being sampled.

Causal language modeling trains models to predict the next token given all previous tokens. This chapter covered the key concepts:

• Autoregressive factorization decomposes sequence probability into a product of conditional probabilities, each predicting one token from its left context
• The CLM objective minimizes cross-entropy loss between predicted and actual next tokens. This gives dense gradient signals from every position
• Causal masking in attention layers enforces the left-to-right information flow, preventing the model from seeing future tokens during training
• Training data quality and scale directly determine model capabilities, with modern LLMs consuming trillions of curated tokens
• Scaling laws show predictable improvements as compute and data, along with parameters, increase, following power-law relationships
• Perplexity measures model quality as the exponential of average loss, with lower values indicating better predictions

The next chapter explores masked language modeling, a bidirectional alternative that trades generation capability for richer contextual representations.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about causal language modeling and next-token prediction.