Special Tokens › The Purpose of Special Tokens
Language AI HandbookAll chapters Special Tokens › The Purpose of Special TokensOne section · Sections in this chapter2 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Special tokens are entries in the vocabulary that are reserved for structural roles rather than linguistic content. Unlike tokens such as ##ing or ▁the, which represent subword units with linguistic meaning, special tokens convey information about the structure of the input: its boundaries, its composition, its missing parts, and its purpose.
Think of special tokens as stage directions in a screenplay. The dialogue itself (the word tokens) conveys meaning, but the stage directions tell the actors where to stand, when to enter, and what they're supposed to be doing. Without stage directions, the actors could still read the lines, but they'd have no reliable way to know whether this scene is a dialogue or a monologue, whether the next character speaks before or after them, or whether a pause is significant. Special tokens play the same role for a transformer model: they provide the stage directions that make the content interpretable.
A vocabulary entry reserved for structural or control purposes rather than linguistic content. Special tokens are added by the tokenizer during preprocessing and are recognized by the model through their unique IDs. They are always treated as atomic units and are never split by subword tokenization algorithms.
The need for special tokens arises directly from the limitations of raw text sequences. When you feed a sentence into a transformer, the model sees a sequence of token IDs. It has no way to know whether position 0 is the start of a new sequence, whether positions 40 through 47 were padded to match the batch length, or whether a particular token was masked for a training objective. Special tokens inject that structural information into the ID stream itself, co-locating the signal with the content it annotates.
A key property that distinguishes special tokens from regular vocabulary entries is their guaranteed atomicity. A word like "summarize" might be tokenized as ["sum", "##mar", "##ize"] by WordPiece, fragmenting the word's identity across three IDs. A special token like <summarize> is always mapped to exactly one ID, regardless of whether its character sequence would otherwise match vocabulary entries. This atomicity is precisely what makes special tokens reliable as structural signals: you can always predict how they will appear in a token ID sequence, making them safe to use as hard boundaries or control signals.
Different model families have developed different conventions for special tokens. This reflects their different pre-training objectives and later task designs. BERT, GPT, T5, and other architectures each have their own set. But the underlying purposes, marking boundaries, indicating padding, signaling masked positions, and providing classification hooks, are universal. Learning to read these conventions is a transferable skill that will help across every transformer architecture you encounter.