Special Tokens › Summary

Special tokens are the control structures of the tokenizer interface. They inject structural information into plain token ID sequences, enabling a single model architecture to handle classification, sequence labeling, generation, and sentence-pair reasoning by changing only how inputs are framed.

The key tokens and their roles are:

• [CLS]: Provides an aggregate input representation for classification tasks; its final hidden state feeds into classification heads after fine-tuning. Use mean pooling instead of raw [CLS] when working with pre-trained models without task-specific fine-tuning.
• [SEP]: Marks sentence or segment boundaries; combined with token type IDs, distinguishes segments in multi-sentence inputs. Even single-sentence inputs require a closing [SEP].
• [PAD]: Fills shorter sequences to enable batching; must be masked in attention computation and excluded from loss calculation using the standard -100 ignore index.
• [MASK]: Enables BERT's MLM pre-training by marking positions the model must predict; the 80/10/10 replacement strategy reduces the training/inference mismatch by ensuring non-[MASK] tokens appear at masked positions during pre-training.
• [UNK]: Handles out-of-vocabulary characters or subwords; rarely seen in well-designed tokenizers with character-level coverage or byte fallback, but worth monitoring as a distribution shift indicator in production systems.

Beyond BERT's vocabulary, <BOS> and <EOS> tokens serve analogous roles in generative models, framing generation start and completion. Token type IDs provide segment-level structural signals via learned segment embeddings, though modern models like RoBERTa have shown these can be dropped without performance loss.

The most important practical guidelines when working with special tokens are: always use a model's corresponding tokenizer to ensure ID assignments are consistent; verify that padding is masked in attention and excluded from loss computation; when adding custom special tokens, resize the model's embedding layer before fine-tuning and consider mean-of-related-tokens initialization; and when working with sentence embeddings, fine-tune [CLS] for your specific similarity task rather than relying on its raw pre-trained state.

The next chapter examines the practical challenges that arise when tokenizers encounter difficult inputs: numbers, code, multilingual text, emoji, and adversarial inputs designed to exploit tokenization artifacts.