1948: Shannon's N-gram Model

Claude Shannon's 1948 paper, "A Mathematical Theory of Communication", introduced the concept of n-gram models—a foundational idea in natural language processing. Although Shannon's main focus was information theory, his work laid the groundwork for statistical language modeling.

What Is an N-gram, and Why Does It Matter?

At its core, an n-gram is a short sequence of words taken from a text:

• Unigram: a single word
• Bigram: two words in a row
• Trigram: three words in a row

Shannon realized that language isn't random. Certain words are much more likely to follow others. For example, after the word "peanut," the word "butter" is more likely than "giraffe." By looking at lots of text and counting which word sequences appear most often, we can start to predict what comes next.

To show how predictable language can be, Shannon asked people to guess the next letter in a sentence, given the letters so far. He found that people could often make good guesses, especially when there was enough context. This showed that language has patterns, and that some parts are easier to predict than others. The less predictable a word or letter is, the more "information" it carries.

How N-gram Models Work and Where They're Used

The basic idea behind n-gram models is simple. To guess the next word, just look at the last few words. For example, if you see "peanut butter and," you might guess "jelly" comes next. The model doesn't try to understand the meaning. It just relies on how often certain word combinations appear together in real text. This approach is sometimes called the "Markov assumption," meaning the model only cares about the recent past, not the whole sentence.

N-gram models became the backbone of many early language technologies, including:

• Language modeling: Helping computers guess what word comes next in a sentence (useful for autocomplete or grammar checking)
• Machine translation: Helping translation systems choose the most natural-sounding word order in the target language
• Speech recognition: Helping computers decide which word sequence makes the most sense when turning spoken words into text

Challenges, Improvements, and Lasting Impact

As people used n-gram models, they ran into a problem. Many word combinations never appear in the training data, even though they're possible. In 1987, Slava Katz introduced the "Katz back-off" method. If the model hasn't seen a long word sequence before, it "backs off" and looks at shorter ones instead, making the model more flexible and less likely to get stuck.

Despite their usefulness, n-gram models have some big drawbacks:

• Limited memory: They only look at a few words at a time, so they can't capture long-range connections in language
• No real understanding: They don't know what words mean, just which ones tend to go together
• Data hunger: As you look at longer word sequences, you need much more data to see all the possibilities
• Storage: Keeping track of all possible n-grams can take up a lot of space

Even with these limitations, n-gram models are still important. They're easy to understand and explain, work surprisingly well for many simple tasks, are fast and efficient for small-scale problems, and help us measure how well a computer predicts language.

From N-grams to Modern Language Models

Today's language models are much more powerful, but n-grams haven't disappeared. They're still used in some specialized applications and remain a great way to learn the basics of how computers process language.

Because n-gram models can't capture deeper meaning or long-distance relationships in language, researchers developed new methods. Modern neural network models can remember much more context, understand subtle patterns, and even learn the meanings of words. But the basic idea—using patterns in real text to make predictions—started with Shannon's n-gram model.

Shannon's simple insight laid the foundation for decades of progress in language technology, and the n-gram model remains a key stepping stone in the story of language AI.