N-grams › Skip-grams: Flexible Context Windows
Language AI HandbookAll chapters N-grams › Skip-grams: Flexible Context WindowsOne section · Sections in this chapter8 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Standard n-grams require strict adjacency: every token must immediately precede the next one in the sequence. This constraint is linguistically motivated for many phrase-level patterns, but it misses relationships between words that are related despite not being immediately adjacent.
Consider the sentence "the big brown bear ate salmon." The relationship between "bear" and "ate" is meaningful, but they are separated by two words. A standard bigram representation misses this. A trigram captures "big brown bear" and "brown bear ate" but still does not directly pair "bear" with its verb "ate." Skip-grams relax the adjacency constraint to capture these longer-range dependencies.
A skip-gram is a generalization of n-grams that allows gaps between the selected tokens. A -skip--gram selects tokens from a window such that no two consecutive selected tokens are more than positions apart. When , skip-grams reduce to ordinary n-grams. Larger values capture relationships between words that are not immediately adjacent.
The term "skip-gram" in the context of this chapter refers to this linguistic extension. Note that the word "skip-gram" also names a Word2Vec training objective (covered in a later chapter), but the two uses are related: the Word2Vec skip-gram model was inspired by this linguistic notion of flexible context windows.
def extract_skip_bigrams(tokens, max_skip):
"""Extract skip-bigrams with up to max_skip words between."""
skip_bigrams = []
for i in range(len(tokens)):
for j in range(i + 1, min(i + max_skip + 2, len(tokens))):
skip_bigrams.append((tokens[i], tokens[j]))
return skip_bigrams
# Compare regular bigrams with skip-bigrams
sentence = "the cat sat on the mat"
tokens = sentence.split()
regular_bigrams = extract_ngrams(tokens, 2)
skip_1_bigrams = extract_skip_bigrams(tokens, 1)
skip_2_bigrams = extract_skip_bigrams(tokens, 2)Sentence: 'the cat sat on the mat'
Regular bigrams (5):
[('the', 'cat'), ('cat', 'sat'), ('sat', 'on'), ('on', 'the'), ('the', 'mat')]
Skip-1 bigrams (9):
[('the', 'cat'), ('the', 'sat'), ('cat', 'sat'), ('cat', 'on'), ('sat', 'on'), ('sat', 'the'), ('on', 'the'), ('on', 'mat'), ('the', 'mat')]
Skip-2 bigrams (12):
[('the', 'cat'), ('the', 'sat'), ('the', 'on'), ('cat', 'sat'), ('cat', 'on'), ('cat', 'the'), ('sat', 'on'), ('sat', 'the'), ('sat', 'mat'), ('on', 'the'), ('on', 'mat'), ('the', 'mat')]With skip-1 bigrams, we capture pairs like ("cat", "on") and ("sat", "the") that regular bigrams miss entirely. With skip-2 bigrams, we reach even further, capturing ("the", "on") and ("cat", "the"). The total number of pairs grows quadratically with the skip distance, since each word can now be paired with multiple downstream words rather than just the immediately following one.
Skip-grams are useful in several scenarios:
- Word order variation: Different phrasings of the same idea may use the same words in different positions. Skip-grams make the representation more invariant to minor reorderings.
- Intervening modifiers: "The big fluffy cat" and "the cat" both express a relationship between "the" and "cat" that skip-bigrams capture regardless of how many modifiers intervene.
- Long-distance verb-object dependencies: In "she carefully and deliberately chose the option," "chose" and "option" are separated by several words. Skip-bigrams can capture this relationship.
The trade-off, as always, is vocabulary size. Skip-grams with skips produce substantially more combinations than ordinary bigrams:
# Count vocabulary sizes for different skip values
skip_vocab_sizes = {}
for max_skip in range(4):
if max_skip == 0:
all_skipgrams = []
for doc in corpus:
doc_tokens = doc.lower().split()
all_skipgrams.extend(extract_ngrams(doc_tokens, 2))
else:
all_skipgrams = []
for doc in corpus:
doc_tokens = doc.lower().split()
all_skipgrams.extend(extract_skip_bigrams(doc_tokens, max_skip))
skip_vocab_sizes[max_skip] = len(set(all_skipgrams))Vocabulary size by skip distance: ---------------------------------------- Regular bigrams 15 unique Skip-1 bigrams 29 unique Skip-2 bigrams 38 unique Skip-3 bigrams 44 unique
