N-grams › Impact and Applications

The impact of n-gram methods on the history of language technology is hard to overstate. Before neural networks became practical in the late 2000s and early 2010s, n-gram language models were the dominant approach for any application requiring statistical knowledge about word sequences.

Language modeling and speech recognition was the original application and remained dominated by n-gram models for three decades. IBM's trigram models from the 1980s reduced word error rates in dictation systems by more than half compared to earlier methods. Google's voice search and Apple's Siri relied on n-gram language models as core components well into the 2010s.

Statistical machine translation used n-gram language models as one of two key components (alongside a translation model). The language model ensured that the target-side output was fluent English (or whatever target language), scored by a trigram or 5-gram model trained on billions of words of text. This approach produced the first practically useful machine translation systems.

Text classification benefits substantially from adding bigrams to a bag-of-words feature set. In spam detection, sentiment analysis, and topic classification, bigrams capture phrases that single words cannot. The bigram "not good" carries entirely different sentiment from either "not" or "good" individually.

Spell checking and autocomplete both rely on character n-grams and word n-grams respectively. Character n-gram similarity identifies likely corrections for misspelled words. Word n-grams predict likely next words given the current context, which is the core of every autocomplete system.

Plagiarism detection uses n-gram fingerprinting, also called "shingling." A document is represented as the set of its k-grams (typically with k around 5 to 10 words). Two documents that share many k-grams are likely to contain copied passages. This approach tolerates minor paraphrasing and word substitutions because it requires matching sequences of five or more consecutive words, which is hard to achieve by coincidence.

Information retrieval and phrase-based search use bigram and trigram indices to answer phrase queries efficiently. Searching for "New York Times" as a phrase requires matching the trigram instead of checking only for all three words somewhere in the document.

The transition from n-gram models to neural language models beginning around 2010 to 2013 did not eliminate n-grams from the field. Modern subword tokenizers like BPE (Byte Pair Encoding) and WordPiece operate by iteratively merging the most frequent character n-grams until a target vocabulary size is reached. In a sense, BPE learns which character sequences deserve to be treated as atomic units, automating the process of vocabulary construction that n-gram methods handled by counting. Understanding n-grams provides direct intuition for why these newer tokenization methods work: they are learning the most informative character-level sequence patterns from data, rather than treating every possible sequence uniformly.