CBOW Model › The Core Idea: Predicting Words from Context
Language AI HandbookAll chapters CBOW Model › The Core Idea: Predicting Words from ContextOne section · Sections in this chapter2 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Think about the fill-in-the-blank task: "The quick brown ___ jumps over the lazy dog." Given the surrounding words, you can immediately guess "fox." Your brain combines information from all the context words together and uses the collective signal to identify what fits in the middle. CBOW does the same thing computationally.
The CBOW model learns word representations by training a neural network to predict a center (target) word from its surrounding context words. All context word embeddings are averaged into a single representation before making the prediction. The learned embedding matrix becomes the word embeddings.
The name "Continuous Bag of Words" carries a specific meaning. "Continuous" refers to the use of continuous-valued dense vectors rather than discrete one-hot encodings. "Bag of Words" signals that the model treats context as an unordered collection: it does not distinguish between a word appearing two positions to the left and one appearing one position to the right. Both are simply "context words." This positional indifference is a deliberate design choice that simplifies the model and reduces the number of parameters needed.
This design philosophy connects directly to the distributional hypothesis that underlies all word embedding methods. The hypothesis, sometimes paraphrased as "you shall know a word by the company it keeps," holds that words with similar meanings appear in similar contexts. CBOW operationalizes this idea by forcing the model to reconstruct a word from a compressed representation of its neighborhood. If the model learns to do this well, the embedding of each context word must carry information about the kinds of targets it surrounds. After training, those embeddings capture distributional similarity in a dense, continuous vector space.
The key question is: why does averaging the context embeddings work? At first glance, it seems aggressive to collapse four distinct word vectors into a single centroid. The answer lies in how the training signal accumulates across examples. Consider that "fox" appears surrounded by words like "quick," "brown," "runs," "clever," and "jumped" across thousands of sentences in a large corpus. Each training example shifts the embeddings of those context words slightly toward a region of the embedding space associated with fox-like targets. After many such updates, the centroid of those context words consistently points toward the region where "fox" lives. The averaging operation, far from losing information, acts as a kind of implicit regularization: it forces the model to find embeddings that work well collectively rather than relying on any single context word.
# Demonstrating CBOW's prediction task
sentence = "The quick brown fox jumps over the lazy dog"
words = sentence.lower().split()
def get_cbow_example(words, target_idx, window_size=2):
"""Generate a CBOW training example: (context_words, target_word)."""
target = words[target_idx]
context = []
for offset in range(-window_size, window_size + 1):
if offset == 0:
continue
context_idx = target_idx + offset
if 0 <= context_idx < len(words):
context.append(words[context_idx])
return context, target
# Get CBOW example for predicting "fox" (index 3)
context, target = get_cbow_example(words, target_idx=3, window_size=2)CBOW Training Example ============================================= Sentence: 'The quick brown fox jumps over the lazy dog' Target word: 'fox' (position 3) Window size: 2 Context words: ['quick', 'brown', 'jumps', 'over'] Task: predict 'fox' from ['quick', 'brown', 'jumps', 'over']
This single training example contains one prediction task using four context words. Compare this to Skip-gram: for the same sentence position, Skip-gram would generate four separate training pairs: (fox, quick), (fox, brown), (fox, jumps), and (fox, over). CBOW collapses these into one example. This difference in data efficiency affects training speed in a fundamental way, which we discuss in depth later.
Generating All CBOW Training Examples
A training corpus is converted into CBOW examples by sliding a window over every position in every sentence. For each position, the center word becomes the target and the words within the window on each side become the context.
The window-based approach encodes an important assumption: words that are closer in a sentence are more relevant to predicting the center word than words that are far away. The window size controls how wide the context is. A small window (size 1 or 2) captures tight syntactic relationships: adjectives next to nouns, verbs near their subjects. A larger window (size 5 or more) captures broader topical co-occurrence: words that appear in the same paragraph because they relate to the same subject. The optimal window size depends on what you want your embeddings to capture.
def generate_cbow_pairs(words, window_size=2):
"""Generate all CBOW (context, target) training pairs from a sequence of words."""
pairs = []
for target_idx in range(len(words)):
context = []
for offset in range(-window_size, window_size + 1):
if offset == 0:
continue
context_idx = target_idx + offset
if 0 <= context_idx < len(words):
context.append(words[context_idx])
if context: # Only include positions with at least one context word
pairs.append((context, words[target_idx]))
return pairs
all_pairs = generate_cbow_pairs(words, window_size=2)Sentence: 'The quick brown fox jumps over the lazy dog' Window size: 2 All CBOW training pairs (9 total): ------------------------------------------------------------ [quick, brown ] -> 'the' [the, brown, fox ] -> 'quick' [the, quick, fox, jumps ] -> 'brown' [quick, brown, jumps, over ] -> 'fox' [brown, fox, over, the ] -> 'jumps' [fox, jumps, the, lazy ] -> 'over' [jumps, over, lazy, dog ] -> 'the' [over, the, dog ] -> 'lazy' [the, lazy ] -> 'dog'
Notice that positions near the edges of the sentence have fewer context words. The model handles this naturally: those positions contribute fewer context embeddings to average, but the averaging still works. Words at the sentence edges are neither ignored nor given special treatment. The averaging simply operates over whatever context words are available, and the model learns to work with variable-size contexts.
This flexibility is a practical advantage. Unlike models that require fixed-length inputs, CBOW naturally accommodates the varying context sizes that arise at sentence boundaries. There is no padding required and no masking mechanism: the forward pass simply averages the embeddings of however many context words are present.