Special Tokens › Limitations and Practical Implications

Special tokens elegantly solve the problem of communicating structure to a model without architectural changes, but they carry several limitations worth understanding. These limitations are not obscure edge cases: they affect every practitioner who deploys transformer-based models in production.

The pre-training/fine-tuning mismatch for [MASK] is the most documented issue. The token that is central to BERT's pre-training objective is entirely absent during downstream use. The fine-tuned model is never exposed to [MASK] except during pre-training, yet the pre-training process sees [MASK] at 12% of positions on average (15% selected, of which 80% become [MASK]). This asymmetry means the model's representation of non-masked positions during pre-training is subtly different from its representation during fine-tuning: in pre-training, the model "knows" it might need to reconstruct any position (since any position could be masked in a different batch), while in fine-tuning, all positions are real. This is why models like SpanBERT (which replaces contiguous spans) and XLNet (which uses permutation language modeling without [MASK]) achieve better downstream performance on many tasks: they reduce or eliminate the pre-training/fine-tuning gap.

Padding introduces a practical inefficiency that becomes severe with variable-length text. In a batch where sequences vary widely in length (which is common with real-world text), a large fraction of positions may be padding. This wasted computation scales quadratically with padding length in attention (every real token attends to all positions, including padded ones, before masking). Techniques like "dynamic padding" (padding only to the longest sequence in each batch rather than a global maximum) and "packing" (concatenating multiple short examples into a single long sequence, separated by [SEP] tokens) address this. Packing requires careful attention masking to prevent examples from attending to each other across the [SEP] boundary, which adds implementation complexity. The reward is substantial: packing can reduce training compute by 50% or more when the average sequence length is much shorter than the maximum.

The [CLS] representation for sentence embeddings is often misunderstood, and this misunderstanding has real consequences. Many practitioners use raw BERT's [CLS] output for semantic similarity tasks without fine-tuning. This approach consistently underperforms because BERT's pre-training objective (next-sentence prediction) does not train [CLS] for general semantic equivalence. The NSP task asks a coarse yes/no question about sentence adjacency, not a graded semantic similarity question. Models like Sentence-BERT explicitly fine-tune with a contrastive sentence similarity objective, using labeled sentence pairs, to produce high-quality [CLS]-based embeddings. This fine-tuning step is not optional if you want reliable semantic search. The original Sentence-BERT paper showed that raw BERT [CLS] embeddings performed worse than averaging random GloVe embeddings for semantic textual similarity, a result that should calibrate expectations about what pre-training alone provides.

Custom special tokens are useful but require careful management across the model lifecycle. If you add a [ENGLISH] token and save the tokenizer, you must also save the resized model. This ensures the embedding rows for the new tokens are included. Loading the base pre-trained model without the resize will silently produce embedding lookups for out-of-range IDs, which in many deep learning frameworks wraps around or triggers undefined behavior. The model and tokenizer must always be saved and loaded as a matched pair when custom tokens have been added.

The diversity of special token conventions across model families creates friction when combining components from different ecosystems. A BERT tokenizer's [SEP] and a T5 tokenizer's </s> play similar structural roles but have different IDs, different embedding spaces, and different positional conventions. When porting a fine-tuned head from one base model to another, the special token handling must be explicitly reconfigured, and the fine-tuned head may need to be retrained because the [CLS]-equivalent representation from the new model carries different information. These cross-model compatibility issues are a recurring practical challenge in production NLP systems that mix components from different pre-trained checkpoints.

Finally, custom special tokens in chat models introduce a security surface that is easy to overlook. If a model uses [INST] to mark user turns, a user who knows this convention could attempt to inject [/INST] into their input to prematurely close their turn and inject artificial assistant text. Defense against this attack typically involves either sanitizing user inputs to remove special token strings or using token sequences that are unlikely to appear in normal text. Some systems run the user input through the tokenizer and check for unexpected special token IDs before passing the input to the model. This is not a hypothetical concern: prompt injection exploiting turn-boundary tokens has been demonstrated in practice on deployed chat systems.