Special Tokens › Special Token Embeddings

Special tokens receive their own rows in the embedding matrix, just like regular vocabulary tokens. However, their embeddings have unusual properties compared to word tokens. Understanding these properties helps explain some otherwise puzzling behaviors of BERT-based models.

Unlike word tokens, which learn embeddings that encode linguistic and semantic content (semantic similarity, syntactic role, distributional context), special token embeddings encode structural roles. The [CLS] embedding starts as a random initialization and learns during fine-tuning to be a useful "summarization query." By the end of fine-tuning, the [CLS] embedding occupies a region of embedding space associated with "global summary of the input," shaped by whatever task was used for fine-tuning. Different fine-tuning tasks will pull the [CLS] embedding toward different regions, which is why [CLS] representations are not transferable across tasks without re-fine-tuning.

The [PAD] embedding is almost never receives a substantial update, since padding positions are masked out during attention and excluded from loss computation. Their gradients are effectively zero throughout training. As a result, the [PAD] embedding tends to remain close to its random initialization, occupying a somewhat arbitrary location in embedding space. This is not a problem: since [PAD] embeddings are never attended to (due to the attention mask) and never contribute to loss (due to the label mask), their location in embedding space doesn't affect model behavior.

The [MASK] embedding is interesting: it's used heavily during pre-training but never during fine-tuning or inference on real tasks. The [MASK] token appears at approximately 15% of positions in every pre-training batch, so its embedding receives large gradient signal during pre-training. The model must learn to look past [MASK] and use surrounding context instead of the masked token's embedding. Some researchers have noted that the [MASK] embedding occupies a distinct region of embedding space, clearly separated from all word token embeddings. This reflects its purely structural rather than linguistic role. This separation makes sense: the model has learned that [MASK] means "ignore my embedding identity and use context instead," which naturally pushes the [MASK] embedding away from any region occupied by real words.

When fine-tuning with adapter layers or parameter-efficient methods, the embeddings of special tokens are typically not frozen even when other token embeddings are. The fine-tuning objective often depends critically on specific special tokens (the [CLS] representation for classification) and their embeddings need to participate in gradient updates. Freezing the [CLS] embedding while fine-tuning the classification head would prevent the [CLS] embedding from adapting to the task, significantly degrading performance. This is a practical point worth remembering when designing parameter-efficient fine-tuning schemes.