N-grams › Key Functions and Parameters

When working with n-grams in Python, these are the essential functions and their most important parameters:

nltk.ngrams(sequence, n, pad_left=False, pad_right=False)

• sequence: The input tokens (list of words or characters)
• n: The size of the n-gram (2 for bigrams, 3 for trigrams, etc.)
• pad_left, pad_right: Whether to add padding symbols at boundaries. Useful for language modeling where start and end context matters

nltk.bigrams(sequence) and nltk.trigrams(sequence)

• Convenience functions equivalent to ngrams(sequence, 2) and ngrams(sequence, 3)
• Return generators, so wrap in list() if you need to iterate multiple times

nltk.everygrams(sequence, min_len=1, max_len=-1)

• min_len: Minimum n-gram size to include
• max_len: Maximum n-gram size to include (-1 means use the full sequence length)
• Useful when you want to combine multiple n-gram orders in a single feature set

collections.Counter(iterable)

• Essential for counting n-gram frequencies from any iterable of n-gram tuples
• .most_common(n): Returns the n most frequent items as (item, count) pairs
• Supports arithmetic operations for combining counts across documents

Custom extraction parameters:

• max_skip: For skip-grams, controls how many words can be skipped between selected words. Higher values capture more distant relationships at the cost of larger vocabulary
• pad_symbol: The token used for boundary padding (commonly <s>, </s>, or <PAD>)

Scikit-learn integration:

• CountVectorizer(ngram_range=(1, 2)): Extracts both unigrams and bigrams as a combined feature vector
• TfidfVectorizer(ngram_range=(1, 3)): Extracts unigrams through trigrams with TF-IDF weighting
• analyzer='char': Switches from word n-grams to character n-grams, useful for noisy text or morphologically rich languages