Neural Probabilistic Language Model (2003) › The Problem: The Limitations of N-Gram Models

The traditional approach to language modeling, which had dominated the field for decades, relied on n-gram models that estimated the probability of a word given its context by counting occurrences in training data. These models operated on a simple principle: to predict the next word in a sequence, look at the previous $n-1$ words and count how often each candidate word follows that particular context in the training corpus. A trigram model, for example, would use the previous two words to predict the third, calculating probabilities like $P(\text{word}_3 | \text{word}_1, \text{word}_2)$ by counting occurrences in the training data.

These models used discrete, sparse representations where each word was represented by a unique index or one-hot vector. In a vocabulary of 100,000 words, each word would be represented by a 100,000-dimensional vector with a single 1 in the position corresponding to that word and zeros everywhere else. This representation scheme made it impossible to capture similarities between words. The words "cat" and "dog" were treated as completely different entities, even though they share many semantic properties. Similarly, "happy" and "joyful" were treated as unrelated, despite being near-synonyms.

The curse of dimensionality presented a fundamental challenge for n-gram models. As the context window increased, the number of possible n-gram combinations grew exponentially. A trigram model with a vocabulary of 10,000 words would need to estimate probabilities for $10,000^3 = 10^{12}$ possible trigram combinations. Even with large training corpora, most trigrams would never appear, leading to data sparsity where the model had no information about many valid word sequences. The problem became even more severe for longer n-grams, making it practically impossible to model long-range dependencies in language.

Additionally, n-gram models could not capture semantic relationships between words. Words with similar meanings were treated as completely different entities, preventing the model from generalizing what it learned about one word to semantically related words. If the training data contained the phrase "I like cats" but not "I like dogs," an n-gram model would assign zero or very low probability to the latter sequence, even though dogs and cats are semantically similar in this context. This inability to generalize limited the model's ability to handle rare words, out-of-vocabulary terms, and novel word combinations.

These limitations meant that n-gram models required enormous amounts of training data to achieve reasonable performance, and they still struggled with generalization and handling unseen word sequences. Researchers recognized that a fundamentally different approach was needed, one that could capture the semantic relationships between words and learn more generalizable patterns from data.