N-grams › Character N-grams for Noisy Text
Language AI HandbookAll chapters N-grams › Character N-grams for Noisy TextOne section · Sections in this chapter7 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Word n-grams assume clean, correctly spelled text. But real-world text contains typos, spelling variations, invented words, and domain-specific jargon that may not appear in any vocabulary. Character n-grams handle these issues by operating on character sequences instead of word boundaries.
The key property of character n-grams is that similar words share many character n-grams. "Learning" and "learner" share the character trigrams "lea," "ear," "arn," and "rni." A typo like "lerning" still shares most trigrams with "learning." This makes character n-grams naturally tolerant of surface variation while remaining sensitive to lexical identity.
A character n-gram is a contiguous sequence of n characters from a text, including spaces and punctuation. Character n-grams operate below the word level and can capture subword patterns such as prefixes, suffixes, and stems. They are particularly useful for morphologically rich languages, informal text with spelling variations, and tasks involving noisy input.
def extract_char_ngrams(text, n):
"""Extract character n-grams from text."""
return [text[i : i + n] for i in range(len(text) - n + 1)]
# Compare word and character n-grams
word = "learning"
char_bigrams = extract_char_ngrams(word, 2)
char_trigrams = extract_char_ngrams(word, 3)
# Show how typos affect matching
correct = "learning"
typo = "lerning" # Missing 'a'
correct_trigrams = set(extract_char_ngrams(correct, 3))
typo_trigrams = set(extract_char_ngrams(typo, 3))
overlap = correct_trigrams & typo_trigramsWord: 'learning' Character bigrams: ['le', 'ea', 'ar', 'rn', 'ni', 'in', 'ng'] Character trigrams: ['lea', 'ear', 'arn', 'rni', 'nin', 'ing'] Typo robustness example: Correct: 'learning' -> trigrams: ['arn', 'ear', 'ing', 'lea', 'nin', 'rni'] Typo: 'lerning' -> trigrams: ['ern', 'ing', 'ler', 'nin', 'rni'] Overlap: ['ing', 'nin', 'rni'] (3/6 = 50%)
Despite the missing letter, most character trigrams still match between the correctly spelled word and the typo. This partial matching property is why character n-grams excel at several tasks:
- Spelling correction: Finding the closest dictionary word to a misspelled query by measuring character n-gram overlap
- Language identification: Different languages have characteristic character-level frequency patterns that persist even in short text snippets
- Author attribution: Writing style appears in subtle character-level patterns, from punctuation and spacing to how words are constructed
- Morphologically rich languages: Languages with complex inflectional systems (Turkish, Finnish, Arabic) express grammatical information through affixes; character n-grams capture these subword units naturally
- Social media and informal text: Character n-grams handle usernames and hashtags as well as abbreviations or unconventional spellings better than word-level models that depend on a fixed vocabulary
The trade-off is a much larger character n-gram vocabulary relative to the number of character types. English has only about 40 distinct characters (letters, common punctuation, and space), so character unigrams are tiny. But character trigrams can number in the thousands for natural text, and the full range of possible character 4-grams is .
# Language identification using character trigrams
samples = {
"English": (
"The sun rises early in the morning and sets late in the evening. "
"Children play in the streets while their parents work in the garden. "
"The market opens every morning with fresh fruits and vegetables. "
"People gather together to share stories and enjoy the afternoon sun."
),
"French": (
"Le soleil se lève tôt le matin et se couche tard le soir. "
"Les enfants jouent dans les rues pendant que leurs parents travaillent. "
"Le marché ouvre chaque matin avec des fruits et des légumes frais. "
"Les gens se rassemblent pour partager des histoires et profiter du soleil."
),
"German": (
"Die Sonne geht früh am Morgen auf und spät am Abend unter. "
"Kinder spielen auf den Straßen während ihre Eltern im Garten arbeiten. "
"Der Markt öffnet jeden Morgen mit frischem Obst und frischem Gemüse. "
"Die Menschen versammeln sich um Geschichten zu teilen und die Sonne zu genießen."
),
"Spanish": (
"El sol sale temprano por la mañana y se pone tarde por la noche. "
"Los niños juegan en las calles mientras sus padres trabajan en el jardín. "
"El mercado abre cada mañana con frutas y verduras frescas del campo. "
"La gente se reúne para compartir historias y disfrutar del sol de la tarde."
),
}
# Extract character trigram profiles
profiles = {}
for lang, text in samples.items():
trigrams = extract_char_ngrams(text.lower(), 3)
profiles[lang] = Counter(trigrams).most_common(10)Top 5 character trigrams by language: -------------------------------------------------- English : 'the', 'he ', ' th', ' in', 'in ' French : 'es ', ' le', 'le ', 'ent', ' so' German : 'en ', 'nd ', ' ge', ' un', 'sch' Spanish : 'as ', 'el ', ' la', 'la ', 'l s'
Each language has distinctive character trigram signatures. German shows patterns like "sch" (a common consonant cluster), French has character sequences from its characteristic liaison and nasal vowel patterns, and Spanish shows the influence of its double-letter combinations and accent marks. These patterns form the basis of language detection algorithms, which are fast and accurate even on very short text because character trigram profiles diverge quickly between languages.
