Special Tokens › BERT's Core Special Tokens

BERT popularized a set of special tokens that became a reference point for much of the NLP field. Understanding what each one does, and why BERT needs it, gives you the conceptual framework to understand special token design in any model. BERT's design choices were not arbitrary: each token solves a specific problem that emerges from the architecture's bidirectional training objective and its goal of serving many downstream tasks from a single pre-trained checkpoint.

The pre-training objectives that shaped BERT's special tokens were Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). MLM requires the model to reconstruct masked portions of text using both left and right context, which demands that the model attend bidirectionally to the entire sequence. NSP requires the model to determine whether two sentences naturally follow each other, which demands input representations that can encode a two-sentence pair as a single structured unit. These two objectives together drove the design of all five core special tokens.

The core BERT tokens are:

• [CLS]: Classification token, prepended to every input sequence
• [SEP]: Separator token, appended after every sentence or segment
• [PAD]: Padding token, used to extend shorter sequences to a fixed length
• [MASK]: Mask token, replacing tokens during the Masked Language Model pre-training objective
• [UNK]: Unknown token, substituting for characters or subwords absent from the vocabulary

Let's examine each in detail, because the "why" behind each one is as important as the "what."

The [CLS] token (short for "classification") is prepended to the beginning of every input sequence. Its final hidden state, after passing through all transformer layers, is the aggregate representation of the entire input for classification tasks. Think of [CLS] as a dedicated "read the whole document and summarize it" position: by attending to all other tokens across all transformer layers, it accumulates a compressed representation of the entire sequence by the time it reaches the final layer.

This design was deliberate. In the original BERT paper, Devlin et al. needed a way to extract a single vector representing the whole input sequence for tasks like sentiment classification or next-sentence prediction. A naive approach would be to average all token representations, but simple averaging treats all tokens equally regardless of how informative they are for the task at hand. Instead, they added a dedicated [CLS] position whose representation is computed by attending to all other tokens across all layers. [CLS] can attend to every token with a different attention weight, so it can learn to focus more on the words that matter for the task. This learned, weighted aggregation is strictly more expressive than fixed mean pooling.

The final hidden state of [CLS] is fed into a linear layer with softmax for classification tasks. The model takes the [CLS] hidden state vector and projects it into class scores, which are then normalized into probabilities.

$$P(\text{class} \mid \text{input}) = \text{softmax}(W \cdot \mathbf{h}_{\text{cls}} + \mathbf{b})$$

where:

• $\mathbf{h}_{\text{cls}}$: the hidden state at the [CLS] position after the final transformer layer, a vector of dimension $d$ (the model's hidden size, which is 768 for BERT-base)
• $W$: a learned weight matrix mapping from the hidden dimension $d$ to the number of classes $C$, with shape $[C \times d]$, initialized randomly and trained during fine-tuning
• $\mathbf{b}$: a bias vector of shape $[C]$, one entry per output class

Why does this formula make sense? Notice that $W \cdot \mathbf{h}_{\text{cls}}$ is a dot product between each row of $W$ and the [CLS] vector, producing one scalar score per class. Each row of $W$ can be thought of as a "class direction" in the $d$-dimensional hidden space: a class is predicted as likely when the [CLS] vector points in the same direction as the corresponding row of $W$. The softmax then converts these raw scores into a probability distribution that sums to 1.

The classification probability for each class is the result of projecting the [CLS] hidden state through this linear layer and normalizing with softmax. Because the [CLS] token attends to all other tokens throughout every transformer layer, it accumulates a summary of the entire input sequence by the time it reaches the final layer. During fine-tuning, the classification head is trained alongside the transformer. The [CLS] representation becomes specialized through task-specific gradient updates, learning to aggregate the input in ways useful for the target task. This specialization is why fine-tuning generally outperforms using a frozen pre-trained [CLS] representation: the representation needs to learn what to summarize, and that depends on the task.

One important subtlety: the [CLS] representation is not particularly meaningful without fine-tuning. In pre-trained BERT, the [CLS] vector is trained for next-sentence prediction, which is a weak proxy for general text similarity. The NSP task asks whether two sentences are consecutive in the original document, which is a coarse semantic relationship. If you want to use BERT representations for sentence embeddings without fine-tuning, averaging all non-[PAD] token representations (sometimes called "mean pooling") often works better than taking [CLS] alone. This is a practical implication with significant consequences: many early BERT-based retrieval systems underperformed because they used raw [CLS] embeddings, expecting semantic quality that the pre-training objective had not trained for.

The [SEP] token marks the end of a sentence or segment. In BERT's input format, a pair of sentences A and B is formatted as:

The [SEP] tokens serve two purposes simultaneously. First, they provide a dedicated position at the boundary whose learned representation encodes the end-of-segment signal. Because [SEP] always occupies the boundary position, attention heads can learn to use it as a landmark: any token attending to [SEP] is effectively asking "where does this segment end?" This is structurally similar to how sentence-final punctuation works in natural language: a period doesn't just end a sentence, it signals that what follows belongs to a new unit. Second, combined with token type IDs (discussed in detail later), [SEP] helps the model distinguish which segment each token belongs to.

For single-sentence inputs (such as document classification), only one [SEP] is needed, appended at the end:

Even in the single-sentence case, [SEP] is not optional. It signals to the model that the sequence is complete, preventing the model's attention patterns from treating the last content token as if the sequence might continue. Attention heads trained on data that always ends in [SEP] learn to recognize the end-of-input signal from it.

The [SEP] design reflects BERT's pre-training on two tasks simultaneously: Masked Language Modeling, which operates on individual tokens, and Next Sentence Prediction, which requires the model to reason about two sentences as a pair. The separator makes the boundary explicit. Without [SEP], the model would have to infer sentence boundaries from semantic content alone, which is both harder to learn and less reliable.

Models like RoBERTa, which dropped next-sentence prediction during pre-training, still retained [SEP] for compatibility and because sentence boundaries remain useful structural signals even without an explicit NSP objective. The presence of [SEP] in every pre-training sequence means that any model trained on those sequences has learned to associate [SEP] with "end of meaningful content." Later models took different approaches: GPT-2 uses a single end-of-text token <|endoftext|> for all boundaries. This reflects its autoregressive rather than bidirectional nature. The choice of boundary token is tied to the training objective.

When processing text in batches, sequences of different lengths must be padded to the same length so they can be stacked into a matrix. The [PAD] token fills the unused positions. This is a purely computational necessity: GPUs process batches as fixed-size tensors, and variable-length sequences cannot be efficiently stacked without padding.

Think of padding as the blank spaces on a printed page: they are there to fill the required format (all sequences must be the same width), but they carry no information and should not be read as if they do. If the model were allowed to "read" the blank spaces, it would start extracting spurious meaning from the fact that one sequence was shorter than another, which would corrupt the representations.

Padding is necessary because GPU computation operates on fixed-size tensors. If you have sequences of length 12, 45, and 78 in the same batch, you need to extend the first two to length 78. The [PAD] token fills positions 13 through 78 and 46 through 78 respectively.

The critical detail is that padding tokens must be masked during attention computation. The attention mechanism computes scores between all token pairs in the sequence. If padding tokens participate fully in attention, the model will learn spurious dependencies: real tokens will "attend to" padding tokens, diluting their representations with noise. More importantly, the [PAD] token's representation will flow back into the [CLS] representation used for classification, corrupting the aggregate signal. The longer the padding, the worse this effect.

This is handled through an attention mask: a binary tensor of the same shape as the input sequence, where 1 indicates a real token and 0 indicates a padding token. Before applying softmax, attention scores at padding positions are set to a large negative number (typically $-10^4$ in practice, approximating $-\infty$), effectively zeroing out their softmax probabilities.

The masked attention score computation for a query position attending to all key positions describes how a single query vector $\mathbf{q}$ produces a score for key position $j$:

$$s_j = \frac{\mathbf{q} \cdot \mathbf{k}_j}{\sqrt{d_k}} + m_j$$

where:

• $\mathbf{q}$: the query vector for the current position, shape $[d_k]$, derived by linearly projecting the current position's hidden state
• $\mathbf{k}_j$: the key vector at position $j$, shape $[d_k]$, derived by linearly projecting position $j$'s hidden state
• $d_k$: the key dimension, used for scaling to prevent dot products from growing too large in high-dimensional spaces, which would cause the softmax gradients to become vanishingly small
• $m_j$: the mask value, where $m_j = 0$ for real tokens (no effect on the score) and $m_j = -10^4$ (approximating $-\infty$) for padding tokens

Why does this formula make sense? Notice that $m_j = -\infty$ makes $s_j = -\infty$ for padding positions. When these scores are passed through softmax, the softmax formula $e^{s_j} / \sum_k e^{s_k}$ makes $e^{-\infty} = 0$, so padding positions contribute exactly zero weight to the attention distribution. Real token positions compute attention normally, attending only to other real tokens.

The loss function must also mask out padding positions during pre-training and fine-tuning: you don't want the model penalized for wrong predictions on positions that weren't real tokens to begin with. HuggingFace models typically accept a labels tensor where padding positions are set to -100, which is the standard ignore index for PyTorch's cross-entropy loss. This -100 convention comes directly from the PyTorch loss function's ignore_index parameter, and it means that gradients from padding positions do not flow back through the model.

The [MASK] token is the centerpiece of BERT's Masked Language Modeling (MLM) pre-training objective. It is the token that made BERT's bidirectional pre-training possible. The key insight behind MLM is this: if you want a model to learn bidirectional context, you need a training objective that forces it to use both left and right context simultaneously. But standard language modeling (predict the next token) is inherently left-to-right. MLM breaks this constraint by hiding random tokens and asking the model to reconstruct them using all surrounding context.

During pre-training, 15% of tokens are selected for masking. Of those selected tokens:

• 80% are replaced with [MASK]
• 10% are replaced with a random vocabulary token
• 10% are kept unchanged

The model must predict the original token at every masked position. This forces the model to develop bidirectional contextual representations, since it must use context from both left and right to fill in the missing word.

The reason for the 80/10/10 split is instructive, and the logic behind it is worth understanding carefully. If every selected token were always replaced with [MASK], the model would learn to ignore the identity of tokens at masked positions during fine-tuning (since [MASK] never appears in real text). The model would effectively learn: "whenever I see [MASK], use context to fill it in; whenever I see a real token, don't bother checking if it might be wrong." This produces a model that processes [MASK] positions very differently from regular positions.

By sometimes keeping the original token unchanged, the model is forced to maintain useful representations for every token, not just those at explicit [MASK] positions. The model cannot distinguish which positions were "selected" without being told: it must process every token as if it might need to verify its prediction against the label. By sometimes replacing with a random word, the model learns to handle noisy input: even if a position contains an implausible word, the model should still use surrounding context rather than blindly trusting the token's identity.

There's a well-known limitation called the pre-training/fine-tuning mismatch: [MASK] tokens appear during pre-training but never during fine-tuning on real tasks. The fine-tuned model is applied to natural text that contains no [MASK] tokens, so the model must generalize from a pre-training distribution that includes [MASK] to a fine-tuning distribution that doesn't. The 80/10/10 strategy mitigates this by ensuring the model sees some non-[MASK] tokens in masked positions during pre-training, but the mismatch isn't fully eliminated. This was one motivation for models like SpanBERT, which replaces contiguous spans rather than individual tokens, and XLNet, which uses a permutation-based objective that avoids [MASK] entirely. Understanding this mismatch helps explain why some downstream tasks benefit from continued pre-training on domain text before fine-tuning: the distribution shift between [MASK]-heavy pre-training and clean downstream text can be partially bridged by additional unsupervised training on real text without masking.

The [UNK] token handles cases where a character or subword is not in the vocabulary. This is a defensive token: its role is to prevent the tokenizer from failing on unexpected input by providing a fallback encoding for anything that cannot be represented within the known vocabulary. Think of [UNK] as the "other" category in a classification system: you hope to rarely need it, but you need it as a fallback.

The [UNK] token can appear in several situations:

• The input contains characters outside the character coverage used during tokenizer training (for example, a rare CJK character if the tokenizer was trained on mostly English text)
• The vocabulary was built from a domain that didn't include certain rare tokens
• Input contains encoding errors, unusual Unicode sequences, or control characters that were excluded from training data
• The tokenizer uses a fixed character whitelist that doesn't include every possible Unicode code point

In well-designed subword tokenizers, [UNK] should rarely appear. WordPiece builds its vocabulary from individual characters first. This ensures that any Unicode character that appeared in the training corpus can at minimum be encoded character-by-character. If the tokenizer has seen the individual characters, it can always fall back to character-level encoding for unknown words rather than emitting [UNK]. Similarly, SentencePiece's byte fallback mode encodes unknown characters as sequences of byte tokens (each byte is a valid vocabulary entry), removing [UNK] by making any possible character representable through byte sequences. For this reason, modern tokenizers like Llama's use byte fallback: a [UNK]-free tokenizer never loses information.

Despite its defensive nature, [UNK] can cause subtle issues. When a named entity or technical term hits [UNK], the model loses the lexical signal from that token entirely. All terms that map to [UNK] receive exactly the same embedding: they are indistinguishable from the model's perspective. In high-stakes applications, monitoring the [UNK] rate on your inference data is a useful diagnostic. A spike in [UNK] tokens often indicates that your deployment distribution has shifted away from your training distribution, for example, when a model trained on English text starts receiving inputs with significant non-English content. This monitoring is cheap to implement and can catch distribution shift before it causes visible failures.