FastText › Character N-grams: Representing Words as Substrings
Language AI HandbookAll chapters FastText › Character N-grams: Representing Words as SubstringsOne section · Sections in this chapter3 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The core data structure in FastText is the character n-gram. For any word, FastText first wraps it in boundary markers, then extracts every contiguous substring of characters within a configured length range. The result is a set of overlapping fragments that collectively encode the word's character structure.
A character n-gram is a contiguous sequence of characters within a word. For the word "where," the 3-grams are "whe," "her," and "ere." FastText adds special boundary markers at the start and end of each word, transforming "where" into "<where>" before extraction. This ensures that "<wh" distinguishes a word-initial position from the interior substring "her," which could appear in many different words at different positions.
The function below extracts all n-grams within a length range. This is the literal preprocessing step applied to every word in the training corpus:
def get_ngrams(word, min_n=3, max_n=6):
"""
Extract character n-grams from a word with boundary markers.
The word is wrapped in < and > before extraction so that
prefix n-grams (like '<wh') and suffix n-grams (like 're>')
are distinguished from internal substrings.
"""
marked = f"<{word}>"
ngrams = []
for n in range(min_n, max_n + 1):
for i in range(len(marked) - n + 1):
ngrams.append(marked[i : i + n])
return ngrams
example_word = "where"
ngrams = get_ngrams(example_word, min_n=3, max_n=6)Character n-grams for 'where' (marked as '<where>'): ------------------------------------------------------- 3-grams (5): ['<wh', 'whe', 'her', 'ere', 're>'] 4-grams (4): ['<whe', 'wher', 'here', 'ere>'] 5-grams (3): ['<wher', 'where', 'here>'] 6-grams (2): ['<where', 'where>'] Total n-grams: 14
The word "where" produces 18 n-grams spanning lengths 3 to 6. Notice what these substrings encode. The 3-gram "<wh" can only appear at the very start of a word: the "<" marker ensures this. Similarly, "re>" can only appear at a word ending. The internal 3-gram "her" has no positional constraint, so it can match "hero," "mother," "together," and any other word containing that substring in the middle. This positional disambiguation is one of the key functions of the boundary markers.
The n-gram length range is a deliberate design choice. A minimum of 3 avoids extremely short substrings like "re" or "he" that appear in so many words they carry little discriminative information. A maximum of 6 is long enough to capture most productive English morphemes: the suffix "-ation" is 6 characters, "-ment" is 4, the prefix "pre-" is 3. For languages with longer morphemes, the maximum can be extended.
Why Boundary Markers Matter
Boundary markers do more than look like a clever trick. They fundamentally change which words share n-grams and by how much. Without markers, "where," "hero," and "other" would all share the trigram "her," conflating three very different words in ways that do not reflect their linguistic relationship. With markers, each word develops a distinct positional fingerprint.
# Compare how boundary markers affect n-gram sets
words_with_her = ["where", "hero", "other"]
trigram_sets = {}
for word in words_with_her:
marked = f"<{word}>"
trigrams = {marked[i : i + 3] for i in range(len(marked) - 2)}
trigram_sets[word] = trigramsEffect of boundary markers on trigram overlap:
-------------------------------------------------------
'where' -> '<where>':
Trigrams: ['<wh', 'ere', 'her', 're>', 'whe']
'hero' -> '<hero>':
Trigrams: ['<he', 'ero', 'her', 'ro>']
'other' -> '<other>':
Trigrams: ['<ot', 'er>', 'her', 'oth', 'the']
The raw substring 'her' appears in all three words.
But with markers, each word's trigram set is distinct:
'hero' & 'where' share: ['her']
'hero' & 'other' share: ['her']
'other' & 'where' share: ['her']Without markers, "where," "hero," and "other" would all share the trigram "her," merging their embeddings in a way that does not reflect a linguistic relationship. The "her" inside "other" is a suffix fragment; the "her" in "where" is an internal sequence; the "her" at the start of "hero" is a prefix. These positions carry different information, and boundary markers let FastText model them separately.
The practical implication is significant. Consider morphologically related pairs like "run" and "runner." With boundary markers, "<run>" and "<runner>" share 3-grams "<ru," "run," "un>" (for the shorter word) and "<ru," "run," "unn," "nn," "nne," "ner," "er>" (for the longer word). Several markers align: the prefix fragments reflect that both words start the same way, and the "run" interior fragment captures the shared root. This shared structure is exactly what we want the model to exploit.
N-gram Count Scaling
The number of n-grams a word generates grows with its length. Short words have fewer substrings, which gives them less representational capacity. Long words have more, which helps with morphologically complex forms that pack meaning into their length.
# Measure how n-gram count scales with word length
test_words_by_length = {
3: ["run", "cat", "dog"],
4: ["walk", "fish", "jump"],
5: ["where", "think", "happy"],
6: ["runner", "moving", "faster"],
7: ["running", "walking", "jumping"],
8: ["swimming", "watching", "learning"],
10: ["understand", "completely", "experience"],
12: ["professional", "conversation", "appreciation"],
}
lengths = sorted(test_words_by_length.keys())
avg_counts = []
by_size = {n: [] for n in [3, 4, 5, 6]}
for length in lengths:
words = test_words_by_length[length]
counts = []
size_totals = {n: 0 for n in [3, 4, 5, 6]}
for word in words:
ngs = get_ngrams(word, min_n=3, max_n=6)
counts.append(len(ngs))
for ng in ngs:
if len(ng) in size_totals:
size_totals[len(ng)] += 1
avg_counts.append(np.mean(counts))
for n in [3, 4, 5, 6]:
by_size[n].append(size_totals[n] / len(words))
The roughly linear scaling is important. It means that the number of parameters used to represent a word grows proportionally with the word's length, which aligns well with the intuition that longer words carry more information per token. Short words like "run" (3 characters) get 9 n-grams; longer words like "appreciation" (12 characters) get around 45. This proportionality is not accidental: the architects of FastText chose the extraction range specifically to maintain this relationship between word complexity and representational capacity.
The distribution by n-gram size also matters. Shorter n-grams (3 and 4 characters) appear in many words, so their embeddings are updated frequently during training and tend to capture broad morphological patterns. Longer n-grams (5 and 6 characters) are more specific, appearing in fewer words, so they capture finer-grained structure. Together, the range from 3 to 6 provides both broad pattern recognition and fine-grained discrimination.