Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Skip-gram Model › Training Data Generation

Michael BrenndoerferApril 2, 20255 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 19Skip-gram Model › Training Data Generation

Reading section 6 of 14 · Training Data Generation · 5 minSection 6 of 14 · 5 min

Section 6 of 14 · 5 min read

Skip-gram Model › Training Data GenerationOne section · Sections in this chapter6 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One of Skip-gram's most important strengths is that training data generation is entirely automatic. Any large text corpus becomes training data without manual labeling. The algorithm is:

Tokenize the corpus into words
Slide a window of size $2c + 1$ across each sentence
For each position $t$ , pair the center word with each context word in the window
Each (center, context) pair becomes one training example

This self-supervised nature means you can train on Wikipedia, books, news articles, web crawls, or any other text. The training signal comes entirely from the statistical patterns in the text, with no human annotation needed. This property allows the method to scale: it means you can scale arbitrarily by adding more data, and more data consistently produces better embeddings.

In[6]:

Code

def generate_training_data(corpus, window_size=2):
    """Generate (center, context) training pairs from a text corpus."""
    training_pairs = []
    sentences = corpus.strip().split(".")

    for sentence in sentences:
        words_raw = sentence.strip().lower().split()
        words_clean = [
            w.strip(".,!?;:") for w in words_raw if w.strip(".,!?;:")
        ]
        if len(words_clean) < 2:
            continue
        for i, center_word in enumerate(words_clean):
            for offset in range(-window_size, window_size + 1):
                if offset == 0:
                    continue
                context_idx = i + offset
                if 0 <= context_idx < len(words_clean):
                    training_pairs.append(
                        (center_word, words_clean[context_idx])
                    )

    return training_pairs


corpus = """
The cat sat on the mat. The dog ran in the park.
The cat chased the mouse. The dog barked at the cat.
A dog and a cat can be friends. Cats and dogs both make good pets.
"""

pairs = generate_training_data(corpus, window_size=2)

# Build vocabulary
vocab = sorted(set(word for pair in pairs for word in pair))
word_to_idx = {w: i for i, w in enumerate(vocab)}
idx_to_word = {i: w for w, i in word_to_idx.items()}

Out[7]:

Console

Corpus statistics:
  Vocabulary size: 24
  Total training pairs: 116

Sample training pairs:
  ('the', 'cat')
  ('the', 'sat')
  ('cat', 'the')
  ('cat', 'sat')
  ('cat', 'on')
  ('sat', 'the')
  ('sat', 'cat')
  ('sat', 'on')

Vocabulary (first 15 words):
  a, and, at, barked, be, both, can, cat, cats, chased, dog, dogs, friends, good, in

The training data volume grows quickly. For a corpus of $T$ words with window size $c$ , each word generates up to $2c$ training pairs, yielding up to $2cT$ pairs total. A Wikipedia dump with 2 billion tokens and window size 5 generates up to 20 billion training pairs. This scale is part of what makes Skip-gram work: the model sees each word from many different perspectives, building a rich picture of its contextual distribution.

Why This Data Generation WorksLink Copied

The data generation strategy reflects a key assumption: the meaning of a word can be inferred from the company it keeps, a principle known as the distributional hypothesis. Words that appear near "hospital" include "doctor," "nurse," "patient," "medicine," and "surgery." If we train a model to predict these context words from "hospital," the model learns that the "hospital" vector must have something in common with all of them, encoding the semantic field of medicine.

Words sharing many context words will naturally develop similar embeddings, because they must predict similar distributions. "Doctor" and "physician" appear in nearly identical contexts, so their embeddings converge toward one another during training. "Bank" appears in both financial and riverine contexts, so its embedding ends up reflecting a blend of both usages, positioned between the financial vocabulary cluster and the geographical vocabulary cluster. The geometry of the embedding space directly encodes distributional structure. What we think of intuitively as "meaning" turns out, to a remarkable degree, to be capturable by these distributional patterns.

The model also implicitly learns syntactic structure. Adjectives appear before nouns, so all adjectives tend to appear in adjective-like contexts. Over time, adjective embeddings cluster together in the space, even if no two particular adjectives share frequent co-occurrences. The same happens for verbs, nouns, and prepositions. This latent syntactic structure in the embedding space is not explicitly trained for; it emerges from the distributional patterns in grammatical text.

Subsampling Frequent WordsLink Copied

In practice, the training data generation step includes one more important operation: subsampling of frequent words. High-frequency words like "the," "a," "in," and "of" appear so often that they end up as context words for almost every center word. This creates a lot of uninformative training pairs: knowing that "fox" appeared near "the" tells you almost nothing about the meaning of "fox," because "the" appears near everything.

Mikolov et al. introduced a subsampling heuristic that discards each word $w$ from the training data with probability:

P(\text{discard}) = 1 - \sqrt{\frac{t}{f(w)}}

where $f(w)$ is the frequency of word $w$ in the corpus and $t$ is a threshold (typically around $10^{-5}$ ). Words with frequencies much higher than $t$ are discarded with high probability, while rare words are almost never discarded. The effect is twofold: it speeds up training by generating fewer training pairs, and it improves embedding quality by giving rarer, more informative words a larger share of the gradient signal.