How Language Models Work (in Plain English) › Training: Learning from Examples

How does a language model learn to predict the next word? Through a process called training, which happens long before you ever interact with the model.

Here's the basic idea:

Step 1: Gather massive amounts of text

The model's creators collect a huge dataset: books, articles, websites, code repositories, you name it. This is the model's "reading material." The more diverse and high-quality the text, the better the model can learn.

Step 2: Show the model partial sentences

During training, the model sees a sentence with some words hidden. For example:

The model tries to predict what comes next. At first, it's basically guessing randomly.

Step 3: Correct the model's mistakes

After the model makes a prediction, the training process compares it to the actual next word. If the model guessed "London" instead of "Paris," it gets corrected. The model's internal parameters (think of these as millions of tiny knobs) get adjusted slightly to make "Paris" more likely next time in similar contexts.

Step 4: Repeat billions of times

This process repeats over and over with different examples (millions or billions of them). Gradually, the model's predictions get better. It learns that "The capital of France is" should be followed by "Paris," that "Once upon a" is often followed by "time," and that "2 + 2 =" should be followed by "4."

After training on enough text, the model develops a kind of statistical intuition about language. It hasn't memorized every sentence (there are too many), but it's learned the patterns.

Seeing the same concepts expressed in countless different ways lets the model connect related expressions. It learns that "Paris," "the French capital," and "the city where the Eiffel Tower stands" all refer to the same place. It learns that questions ending with "?" usually expect answers, that code snippets follow specific syntax rules, and that formal writing differs from casual conversation. All of this emerges from the simple task of predicting the next token, repeated billions of times across diverse text.