N-grams › Summary

N-grams extend bag of words by preserving local word order, turning text from a multiset of isolated symbols into a collection of overlapping sequences. Key takeaways:

• N-grams are contiguous sequences of $n$ tokens (words or characters), with unigrams, bigrams, and trigrams as the most common orders
• Bigrams capture immediate word relationships and collocations; trigrams add one more word of context and are standard in language modeling
• Vocabulary explosion means the number of unique n-gram types grows exponentially with $n$, increasing memory use and sparsity while demanding more data
• Zipf's law applies to n-grams at every order, with most types appearing only once and a few appearing extremely often
• N-gram language models use the Markov assumption to approximate the full chain rule of probability, conditioning each word's probability on the preceding $n - 1$ words
• Smoothing is essential in any n-gram language model because maximum likelihood estimation assigns zero probability to unseen n-grams
• Character n-grams tolerate typos and spelling variation while capturing morphological complexity below the word boundary
• Skip-grams relax strict adjacency to capture relationships between non-adjacent words at the cost of larger vocabulary
• N-gram indexing enables fast approximate search and phrase matching through inverted index construction
• Trade-offs are inherent: more context requires more data, creates sparser representations, and makes the zero-count problem worse

N-grams bridge the gap between treating words as isolated units and understanding them in context. In the next chapter, we will explore TF-IDF, which adds statistical weighting to distinguish informative terms from common ones. This provides a complementary approach to improving text representations beyond simple count-based methods.