Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

N-gram Language Models › Start and End Tokens

Michael BrenndoerferMarch 23, 20253 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 8N-gram Language Models › Start and End Tokens

Reading section 5 of 15 · Start and End Tokens · 3 minSection 5 of 15 · 3 min

Section 5 of 15 · 3 min read

N-gram Language Models › Start and End TokensOne section · Sections in this chapter5 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Sentences have beginnings and endings, and a language model needs to capture both. We add special boundary tokens: <s> marks the start of a sentence and </s> marks the end. These tokens allow the model to learn which words typically begin sentences and which words tend to end them.

Without boundary tokens, a bigram model cannot distinguish words that begin sentences from words that appear mid-sentence. If "the" starts every sentence in our corpus, the model should assign high probability to "the" appearing after the sentence boundary. But without a boundary token, there is no special context to condition on for sentence-initial words. The start token <s> provides a synthetic context that captures sentence-initial patterns, enabling the model to compute $P(w_1|\text{<s>})$ , the probability of a word starting a sentence.

The end token </s> is equally important. Without it, the model has no mechanism to learn when sentences typically end. A generation system using the model would need an external stopping criterion. With </s> in the vocabulary, the model learns to assign probability to sentence endings, and generation naturally terminates when </s> is sampled.

In[12]:

Code

def add_sentence_markers(text, n):
    """Add start and end markers to sentences."""
    sentences = text.strip().split("\n")
    marked_sentences = []

    for sentence in sentences:
        if sentence.strip():
            toks = sentence.lower().split()
            # Add n-1 start tokens and 1 end token
            marked = ["<s>"] * (n - 1) + toks + ["</s>"]
            marked_sentences.append(marked)

    return marked_sentences


# Process corpus with markers for bigram model
marked_corpus = add_sentence_markers(corpus, n=2)

# Flatten for counting
all_tokens = [token for sentence in marked_corpus for token in sentence]

# Recount with markers
unigram_counts_marked = count_ngrams(all_tokens, 1)
bigram_counts_marked = count_ngrams(all_tokens, 2)

Out[13]:

Console

Sentences with markers (bigram model):
  1. <s> the cat sat on the mat </s>
  2. <s> the cat ate the fish </s>
  3. <s> the dog sat on the mat </s>
  4. <s> the dog chased the cat </s>

Bigrams involving markers:
  ('<s>', 'the'): 4
  ('</s>', '<s>'): 3
  ('mat', '</s>'): 2
  ('fish', '</s>'): 1
  ('cat', '</s>'): 1

The start marker <s> lets us compute $P(w_1|\text{<s>})$ , the probability of a word starting a sentence. In our corpus, "the" starts every sentence, so $P(\text{the}|\text{<s>}) = 1.0$ . The end marker </s> lets us model sentence endings: "mat," "fish," and "cat" all end sentences.

For higher-order n-grams, we add multiple start tokens. A trigram model needs two <s> tokens at the beginning to provide context for the first two content words. A 4-gram model needs three start tokens. The general rule: prepend $n-1$ start tokens for an order- $n$ model. This ensures that even the very first content words in a sentence have a full context of length $n-1$ available.