Neural Probabilistic Language Model (2003) › The Solution: Distributed Word Representations
History of Language AIAll chapters Neural Probabilistic Language Model (2003) › The Solution: Distributed Word RepresentationsOne section · Sections in this chapter3 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The Neural Probabilistic Language Model addressed these limitations by introducing several key innovations that would become foundational to modern NLP. First, the model learned a distributed representation for each word in the vocabulary, mapping each word to a dense, continuous vector in a high-dimensional space. These word vectors, or embeddings, could capture semantic relationships through their geometric properties. Words with similar meanings would have similar vector representations, and semantic relationships could be represented as geometric relationships in the embedding space. Second, the model used a neural network to compute the probability of the next word given the context, allowing it to learn complex, non-linear relationships between words that n-gram models could not capture.
The shift from discrete to distributed representations represented one of the most important conceptual advances in NLP. Instead of treating words as atomic entities with no internal structure, distributed representations allow words to share components of meaning. When words have similar meanings, their vector representations point in similar directions in the embedding space. This geometric interpretation of meaning would become central to all modern language models, from word2vec to transformers.
The architecture of the Neural Probabilistic Language Model consisted of several key components working together to learn both word representations and language modeling simultaneously. The input layer mapped each word in the context window to its corresponding word vector, which was learned during training rather than being predefined. These word vectors were then concatenated to form a single vector representing the entire context. For example, if the context window was four words and each word vector had 100 dimensions, the concatenated context vector would have 400 dimensions.
This context vector was passed through one or more hidden layers of the neural network, which learned to transform the context representation into a probability distribution over the vocabulary. The hidden layers could capture complex, non-linear interactions between words that simple counting-based methods could not represent. The output layer used a softmax function to ensure that the probabilities for all words in the vocabulary summed to one, creating a proper probability distribution. The model was trained using backpropagation to maximize the likelihood of the training data, adjusting both the word vectors and the neural network weights simultaneously.
The main insight of the model was that the word vectors could be learned jointly with the language model parameters, allowing the model to discover representations that were optimized for the language modeling task. This joint training meant that the word vectors would capture the statistical regularities present in the training data, including semantic relationships, syntactic patterns, and other linguistic phenomena. The learned representations could then be used for other tasks, making them a useful byproduct of the language modeling process.