Special Tokens: CLS, SEP, PAD, MASK, and UNK in Transformers

Subword tokenization gives models a manageable vocabulary of meaningful pieces. But vocabulary alone doesn't tell a model what it's supposed to do with a sequence. Is this text the input to a classifier or just half of a pair? Where does one sentence end and another begin? Which positions should be ignored because the batch had variable-length sequences? What word was deliberately erased for the model to predict?

These questions are answered not by ordinary tokens, but by special tokens: purpose-built vocabulary entries that carry structural and semantic signals rather than word-level meaning. Think of special tokens as punctuation for the machine: just as a period signals the end of a sentence and a question mark changes how you interpret what came before it, special tokens signal to the model how to interpret the structure of the whole input sequence. Without them, a raw sequence of subword IDs is ambiguous in ways that would make multi-task learning nearly impossible.

Special tokens are the grammar of the tokenizer interface. They let a single model architecture serve radically different tasks without changing its weights, simply by changing how the input is framed. A BERT model fine-tuned for question answering uses the same underlying weights as one fine-tuned for sentiment classification. The only difference in how inputs are prepared is the arrangement of special tokens around the content. This architectural elegance is one reason the transformer paradigm scaled so successfully across domains.

Understanding special tokens is essential for two reasons. First, they appear in every modern transformer-based model, and misusing them is one of the most common sources of subtle bugs in NLP applications. A missing attention mask for padding tokens, a double-injected [CLS], or an incorrectly handled segment ID can corrupt model inputs in ways that are hard to detect without careful inspection. Second, their design reveals important decisions about model architecture: why BERT uses [CLS] the way it does, why some models need segment embeddings while others don't, and why padding tokens require special handling during attention. Understanding the reasoning behind these choices makes you a better practitioner, not just someone who can copy tokenization boilerplate.

This chapter covers special tokens: what each one does, why it was designed that way, how models learn from them, and how to handle them correctly in practice. We'll go deep on BERT's core special tokens, trace the evolution toward generative models with their own boundary conventions, and finish with a practical guide to custom special tokens for task-specific models. By the end, you will understand the mechanics and the design philosophy that makes special tokens such a flexible control mechanism.

Special tokens are entries in the vocabulary that are reserved for structural roles rather than linguistic content. Unlike tokens such as ##ing or ▁the, which represent subword units with linguistic meaning, special tokens convey information about the structure of the input: its boundaries, its composition, its missing parts, and its purpose.

Think of special tokens as stage directions in a screenplay. The dialogue itself (the word tokens) conveys meaning, but the stage directions tell the actors where to stand, when to enter, and what they're supposed to be doing. Without stage directions, the actors could still read the lines, but they'd have no reliable way to know whether this scene is a dialogue or a monologue, whether the next character speaks before or after them, or whether a pause is significant. Special tokens play the same role for a transformer model: they provide the stage directions that make the content interpretable.

The need for special tokens arises directly from the limitations of raw text sequences. When you feed a sentence into a transformer, the model sees a sequence of token IDs. It has no way to know whether position 0 is the start of a new sequence, whether positions 40 through 47 were padded to match the batch length, or whether a particular token was masked for a training objective. Special tokens inject that structural information into the ID stream itself, co-locating the signal with the content it annotates.

A key property that distinguishes special tokens from regular vocabulary entries is their guaranteed atomicity. A word like "summarize" might be tokenized as ["sum", "##mar", "##ize"] by WordPiece, fragmenting the word's identity across three IDs. A special token like <summarize> is always mapped to exactly one ID, regardless of whether its character sequence would otherwise match vocabulary entries. This atomicity is precisely what makes special tokens reliable as structural signals: you can always predict how they will appear in a token ID sequence, making them safe to use as hard boundaries or control signals.

Different model families have developed different conventions for special tokens. This reflects their different pre-training objectives and later task designs. BERT, GPT, T5, and other architectures each have their own set. But the underlying purposes, marking boundaries, indicating padding, signaling masked positions, and providing classification hooks, are universal. Learning to read these conventions is a transferable skill that will help across every transformer architecture you encounter.

BERT popularized a set of special tokens that became a reference point for much of the NLP field. Understanding what each one does, and why BERT needs it, gives you the conceptual framework to understand special token design in any model. BERT's design choices were not arbitrary: each token solves a specific problem that emerges from the architecture's bidirectional training objective and its goal of serving many downstream tasks from a single pre-trained checkpoint.

The pre-training objectives that shaped BERT's special tokens were Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). MLM requires the model to reconstruct masked portions of text using both left and right context, which demands that the model attend bidirectionally to the entire sequence. NSP requires the model to determine whether two sentences naturally follow each other, which demands input representations that can encode a two-sentence pair as a single structured unit. These two objectives together drove the design of all five core special tokens.

The core BERT tokens are:

• [CLS]: Classification token, prepended to every input sequence
• [SEP]: Separator token, appended after every sentence or segment
• [PAD]: Padding token, used to extend shorter sequences to a fixed length
• [MASK]: Mask token, replacing tokens during the Masked Language Model pre-training objective
• [UNK]: Unknown token, substituting for characters or subwords absent from the vocabulary

Let's examine each in detail, because the "why" behind each one is as important as the "what."

The [CLS] token (short for "classification") is prepended to the beginning of every input sequence. Its final hidden state, after passing through all transformer layers, is the aggregate representation of the entire input for classification tasks. Think of [CLS] as a dedicated "read the whole document and summarize it" position: by attending to all other tokens across all transformer layers, it accumulates a compressed representation of the entire sequence by the time it reaches the final layer.

This design was deliberate. In the original BERT paper, Devlin et al. needed a way to extract a single vector representing the whole input sequence for tasks like sentiment classification or next-sentence prediction. A naive approach would be to average all token representations, but simple averaging treats all tokens equally regardless of how informative they are for the task at hand. Instead, they added a dedicated [CLS] position whose representation is computed by attending to all other tokens across all layers. [CLS] can attend to every token with a different attention weight, so it can learn to focus more on the words that matter for the task. This learned, weighted aggregation is strictly more expressive than fixed mean pooling.

The final hidden state of [CLS] is fed into a linear layer with softmax for classification tasks. The model takes the [CLS] hidden state vector and projects it into class scores, which are then normalized into probabilities.

where:

• $\mathbf{h}_{\text{cls}}$: the hidden state at the [CLS] position after the final transformer layer, a vector of dimension $d$ (the model's hidden size, which is 768 for BERT-base)
• $W$: a learned weight matrix mapping from the hidden dimension $d$ to the number of classes $C$, with shape $[C \times d]$, initialized randomly and trained during fine-tuning
• $\mathbf{b}$: a bias vector of shape $[C]$, one entry per output class

Why does this formula make sense? Notice that $W \cdot \mathbf{h}_{\text{cls}}$ is a dot product between each row of $W$ and the [CLS] vector, producing one scalar score per class. Each row of $W$ can be thought of as a "class direction" in the $d$-dimensional hidden space: a class is predicted as likely when the [CLS] vector points in the same direction as the corresponding row of $W$. The softmax then converts these raw scores into a probability distribution that sums to 1.

The classification probability for each class is the result of projecting the [CLS] hidden state through this linear layer and normalizing with softmax. Because the [CLS] token attends to all other tokens throughout every transformer layer, it accumulates a summary of the entire input sequence by the time it reaches the final layer. During fine-tuning, the classification head is trained alongside the transformer. The [CLS] representation becomes specialized through task-specific gradient updates, learning to aggregate the input in ways useful for the target task. This specialization is why fine-tuning generally outperforms using a frozen pre-trained [CLS] representation: the representation needs to learn what to summarize, and that depends on the task.

One important subtlety: the [CLS] representation is not particularly meaningful without fine-tuning. In pre-trained BERT, the [CLS] vector is trained for next-sentence prediction, which is a weak proxy for general text similarity. The NSP task asks whether two sentences are consecutive in the original document, which is a coarse semantic relationship. If you want to use BERT representations for sentence embeddings without fine-tuning, averaging all non-[PAD] token representations (sometimes called "mean pooling") often works better than taking [CLS] alone. This is a practical implication with significant consequences: many early BERT-based retrieval systems underperformed because they used raw [CLS] embeddings, expecting semantic quality that the pre-training objective had not trained for.

The [SEP] token marks the end of a sentence or segment. In BERT's input format, a pair of sentences A and B is formatted as:

The [SEP] tokens serve two purposes simultaneously. First, they provide a dedicated position at the boundary whose learned representation encodes the end-of-segment signal. Because [SEP] always occupies the boundary position, attention heads can learn to use it as a landmark: any token attending to [SEP] is effectively asking "where does this segment end?" This is structurally similar to how sentence-final punctuation works in natural language: a period doesn't just end a sentence, it signals that what follows belongs to a new unit. Second, combined with token type IDs (discussed in detail later), [SEP] helps the model distinguish which segment each token belongs to.

For single-sentence inputs (such as document classification), only one [SEP] is needed, appended at the end:

Even in the single-sentence case, [SEP] is not optional. It signals to the model that the sequence is complete, preventing the model's attention patterns from treating the last content token as if the sequence might continue. Attention heads trained on data that always ends in [SEP] learn to recognize the end-of-input signal from it.

The [SEP] design reflects BERT's pre-training on two tasks simultaneously: Masked Language Modeling, which operates on individual tokens, and Next Sentence Prediction, which requires the model to reason about two sentences as a pair. The separator makes the boundary explicit. Without [SEP], the model would have to infer sentence boundaries from semantic content alone, which is both harder to learn and less reliable.

Models like RoBERTa, which dropped next-sentence prediction during pre-training, still retained [SEP] for compatibility and because sentence boundaries remain useful structural signals even without an explicit NSP objective. The presence of [SEP] in every pre-training sequence means that any model trained on those sequences has learned to associate [SEP] with "end of meaningful content." Later models took different approaches: GPT-2 uses a single end-of-text token <|endoftext|> for all boundaries. This reflects its autoregressive rather than bidirectional nature. The choice of boundary token is tied to the training objective.

When processing text in batches, sequences of different lengths must be padded to the same length so they can be stacked into a matrix. The [PAD] token fills the unused positions. This is a purely computational necessity: GPUs process batches as fixed-size tensors, and variable-length sequences cannot be efficiently stacked without padding.

Think of padding as the blank spaces on a printed page: they are there to fill the required format (all sequences must be the same width), but they carry no information and should not be read as if they do. If the model were allowed to "read" the blank spaces, it would start extracting spurious meaning from the fact that one sequence was shorter than another, which would corrupt the representations.

Padding is necessary because GPU computation operates on fixed-size tensors. If you have sequences of length 12, 45, and 78 in the same batch, you need to extend the first two to length 78. The [PAD] token fills positions 13 through 78 and 46 through 78 respectively.

The critical detail is that padding tokens must be masked during attention computation. The attention mechanism computes scores between all token pairs in the sequence. If padding tokens participate fully in attention, the model will learn spurious dependencies: real tokens will "attend to" padding tokens, diluting their representations with noise. More importantly, the [PAD] token's representation will flow back into the [CLS] representation used for classification, corrupting the aggregate signal. The longer the padding, the worse this effect.

This is handled through an attention mask: a binary tensor of the same shape as the input sequence, where 1 indicates a real token and 0 indicates a padding token. Before applying softmax, attention scores at padding positions are set to a large negative number (typically $-10^4$ in practice, approximating $-\infty$), effectively zeroing out their softmax probabilities.

The masked attention score computation for a query position attending to all key positions describes how a single query vector $\mathbf{q}$ produces a score for key position $j$:

where:

• $\mathbf{q}$: the query vector for the current position, shape $[d_k]$, derived by linearly projecting the current position's hidden state
• $\mathbf{k}_j$: the key vector at position $j$, shape $[d_k]$, derived by linearly projecting position $j$'s hidden state
• $d_k$: the key dimension, used for scaling to prevent dot products from growing too large in high-dimensional spaces, which would cause the softmax gradients to become vanishingly small
• $m_j$: the mask value, where $m_j = 0$ for real tokens (no effect on the score) and $m_j = -10^4$ (approximating $-\infty$) for padding tokens

Why does this formula make sense? Notice that $m_j = -\infty$ makes $s_j = -\infty$ for padding positions. When these scores are passed through softmax, the softmax formula $e^{s_j} / \sum_k e^{s_k}$ makes $e^{-\infty} = 0$, so padding positions contribute exactly zero weight to the attention distribution. Real token positions compute attention normally, attending only to other real tokens.

The loss function must also mask out padding positions during pre-training and fine-tuning: you don't want the model penalized for wrong predictions on positions that weren't real tokens to begin with. HuggingFace models typically accept a labels tensor where padding positions are set to -100, which is the standard ignore index for PyTorch's cross-entropy loss. This -100 convention comes directly from the PyTorch loss function's ignore_index parameter, and it means that gradients from padding positions do not flow back through the model.

The [MASK] token is the centerpiece of BERT's Masked Language Modeling (MLM) pre-training objective. It is the token that made BERT's bidirectional pre-training possible. The key insight behind MLM is this: if you want a model to learn bidirectional context, you need a training objective that forces it to use both left and right context simultaneously. But standard language modeling (predict the next token) is inherently left-to-right. MLM breaks this constraint by hiding random tokens and asking the model to reconstruct them using all surrounding context.

During pre-training, 15% of tokens are selected for masking. Of those selected tokens:

• 80% are replaced with [MASK]
• 10% are replaced with a random vocabulary token
• 10% are kept unchanged

The model must predict the original token at every masked position. This forces the model to develop bidirectional contextual representations, since it must use context from both left and right to fill in the missing word.

The reason for the 80/10/10 split is instructive, and the logic behind it is worth understanding carefully. If every selected token were always replaced with [MASK], the model would learn to ignore the identity of tokens at masked positions during fine-tuning (since [MASK] never appears in real text). The model would effectively learn: "whenever I see [MASK], use context to fill it in; whenever I see a real token, don't bother checking if it might be wrong." This produces a model that processes [MASK] positions very differently from regular positions.

By sometimes keeping the original token unchanged, the model is forced to maintain useful representations for every token, not just those at explicit [MASK] positions. The model cannot distinguish which positions were "selected" without being told: it must process every token as if it might need to verify its prediction against the label. By sometimes replacing with a random word, the model learns to handle noisy input: even if a position contains an implausible word, the model should still use surrounding context rather than blindly trusting the token's identity.

There's a well-known limitation called the pre-training/fine-tuning mismatch: [MASK] tokens appear during pre-training but never during fine-tuning on real tasks. The fine-tuned model is applied to natural text that contains no [MASK] tokens, so the model must generalize from a pre-training distribution that includes [MASK] to a fine-tuning distribution that doesn't. The 80/10/10 strategy mitigates this by ensuring the model sees some non-[MASK] tokens in masked positions during pre-training, but the mismatch isn't fully eliminated. This was one motivation for models like SpanBERT, which replaces contiguous spans rather than individual tokens, and XLNet, which uses a permutation-based objective that avoids [MASK] entirely. Understanding this mismatch helps explain why some downstream tasks benefit from continued pre-training on domain text before fine-tuning: the distribution shift between [MASK]-heavy pre-training and clean downstream text can be partially bridged by additional unsupervised training on real text without masking.

The [UNK] token handles cases where a character or subword is not in the vocabulary. This is a defensive token: its role is to prevent the tokenizer from failing on unexpected input by providing a fallback encoding for anything that cannot be represented within the known vocabulary. Think of [UNK] as the "other" category in a classification system: you hope to rarely need it, but you need it as a fallback.

The [UNK] token can appear in several situations:

• The input contains characters outside the character coverage used during tokenizer training (for example, a rare CJK character if the tokenizer was trained on mostly English text)
• The vocabulary was built from a domain that didn't include certain rare tokens
• Input contains encoding errors, unusual Unicode sequences, or control characters that were excluded from training data
• The tokenizer uses a fixed character whitelist that doesn't include every possible Unicode code point

In well-designed subword tokenizers, [UNK] should rarely appear. WordPiece builds its vocabulary from individual characters first. This ensures that any Unicode character that appeared in the training corpus can at minimum be encoded character-by-character. If the tokenizer has seen the individual characters, it can always fall back to character-level encoding for unknown words rather than emitting [UNK]. Similarly, SentencePiece's byte fallback mode encodes unknown characters as sequences of byte tokens (each byte is a valid vocabulary entry), removing [UNK] by making any possible character representable through byte sequences. For this reason, modern tokenizers like Llama's use byte fallback: a [UNK]-free tokenizer never loses information.

Despite its defensive nature, [UNK] can cause subtle issues. When a named entity or technical term hits [UNK], the model loses the lexical signal from that token entirely. All terms that map to [UNK] receive exactly the same embedding: they are indistinguishable from the model's perspective. In high-stakes applications, monitoring the [UNK] rate on your inference data is a useful diagnostic. A spike in [UNK] tokens often indicates that your deployment distribution has shifted away from your training distribution, for example, when a model trained on English text starts receiving inputs with significant non-English content. This monitoring is cheap to implement and can catch distribution shift before it causes visible failures.

Not all models use BERT's [CLS]/[SEP] convention. GPT-style autoregressive models, which generate text left-to-right, use different markers for sequence boundaries. The difference in convention reflects a fundamental difference in how these models process text: BERT reads the entire sequence at once and produces contextual representations for every position, while GPT-style models read left-to-right and produce each output token from the context of all preceding tokens.

The shift from encoder-style to decoder-style models brought a corresponding shift in how sequence boundaries are marked. BERT needs to know where each sentence is because it processes the whole sequence simultaneously: the [CLS] and [SEP] tokens provide spatial landmarks in an already-complete sequence. GPT-style models, by contrast, process sequences one token at a time in order: what they need to know is when to start generating and when to stop. This difference in need produces a difference in token design.

Autoregressive models like GPT use beginning-of-sequence (<BOS>) and end-of-sequence (<EOS>) tokens to mark the start and completion of generated text. These tokens play a role similar to [CLS] and [SEP] in BERT, but the analogy is imperfect: <BOS> and <EOS> are about temporal boundaries in a generation process, while [CLS] and [SEP] are about spatial boundaries in a structured input.

The <BOS> token (sometimes written <s> or <|startoftext|>) primes the model for generation. At inference time, generation begins from a prompt that may or may not include this token explicitly. When present, it signals to the model that what follows is the beginning of a new document or generation context. The model's first prediction is conditioned on everything up to and including <BOS>. During training, every sequence in the training corpus is prepended with <BOS>, so the model learns to associate this token with "the beginning of coherent text."

The <EOS> token (sometimes </s> or <|endoftext|>) signals that generation should terminate. During training, sequences end with <EOS>, teaching the model to predict this token when a sequence is naturally complete. At inference time, generation loops until <EOS> is produced (or a maximum length limit is reached). The key insight is that the model learns during training that <EOS> follows the natural conclusion of a text: a story that has reached its end, an answer that is complete, a list that has been fully enumerated. At inference time, the model applies this learned association to decide when to stop, which is why some models generate coherent text that stops cleanly while others trail off or repeat themselves.

GPT-2 uses a single token <|endoftext|> for both purposes, prepending it to documents during training to distinguish document boundaries in long concatenated training sequences. During generation, the model learns that producing this token signals the end of a coherent piece of text. This dual role (marking both the start of one document and the end of the previous one) is a practical simplification that reflects GPT-2's training on a single concatenated stream of text rather than discrete batched sequences.

Encoder-decoder models like T5 use </s> as both a sentence separator (replacing BERT's [SEP]) and an end-of-sequence marker. This reflects T5's text-to-text framing where all tasks are cast as sequence generation. In T5, the encoder processes the input text ending with </s>, and the decoder generates the output text, also ending with </s>. The same token serves both roles because the architecture treats input and output symmetrically as text strings.

Special tokens are assigned specific IDs in the vocabulary, typically reserved at predictable positions. For example, in the BERT tokenizer:

• [PAD] is assigned ID 0 (allowing zero-initialization of padding tensors, since the zero vector is a natural default)
• [UNK] receives ID 100
• [CLS] receives ID 101
• [SEP] receives ID 102
• [MASK] receives ID 103

These conventions are baked into model configurations and are not interchangeable. When loading a pre-trained model, always use its corresponding tokenizer. Mismatched ID assignments are a common source of silent failures where the model silently processes the wrong signals: if you use a different tokenizer that assigns [CLS] to ID 200, the model will look up embedding row 200 when it expects row 101, producing nonsense embeddings for the most important structural token.

A related pitfall is loading a tokenizer without the corresponding special token configuration. HuggingFace's AutoTokenizer.from_pretrained() handles this correctly, loading both the vocabulary and the special token assignments together. Manual vocabulary construction from raw files does not preserve these assignments unless the configuration files are also loaded.

Beyond the tokens themselves, BERT introduces another mechanism for handling multi-segment inputs: token type IDs (also called segment IDs). Token type IDs answer a question that the token sequence alone cannot: for each position in the input, which sentence does it belong to? This question matters for tasks like natural language inference, question answering, and textual entailment, where the model needs to reason about the relationship between two distinct texts.

Think of token type IDs as jersey numbers in a two-team game: players wearing jersey 0 belong to team A, and players wearing jersey 1 belong to team B. The number doesn't describe what the player does, but it tells the referee (the model) which team's goals the player is working toward. Without this signal, the model has no reliable way to know which part of a two-sentence input is the premise and which is the hypothesis in an inference task.

In BERT's embedding layer, the final embedding for each token is the sum of three components. Before we write the formula, it helps to understand what each component contributes. The token embedding encodes what the token is: its linguistic identity and semantic content. The positional embedding encodes where the token is: its position in the sequence. The segment embedding encodes which sentence the token belongs to: its structural role in the multi-sentence input.

where:

• $\mathbf{e}^{\text{token}}_i$: the token embedding for the $i$-th input token, looked up from a vocabulary embedding matrix of shape $[V \times d]$, where $V$ is the vocabulary size (30,522 for BERT-base) and $d$ is the hidden dimension (768 for BERT-base)
• $\mathbf{e}^{\text{position}}_i$: the positional embedding for position $i$, looked up from a position embedding matrix of shape $[L \times d]$, where $L$ is the maximum sequence length (512 for BERT-base)
• $\mathbf{e}^{\text{segment}}_i$: the segment embedding for token $i$, looked up from a segment embedding matrix of shape $[2 \times d]$, with row 0 for sentence A tokens and row 1 for sentence B tokens

Why does this formula make sense? Notice that all three embedding vectors have the same dimension $d$, so their element-wise sum produces a single embedding vector $\mathbf{e}_i$ of dimension $d$. This sum encodes three orthogonal kinds of information simultaneously: what the token is, where it is, and which segment it belongs to. The model can potentially disentangle these signals across different dimensions of the embedding space. The combined embedding is then passed into the first transformer layer, carrying all three kinds of structural information into every subsequent attention computation.

The segment embedding lookup table has only two rows: one for segment A (type ID 0) and one for segment B (type ID 1). The model learns these two vectors during pre-training as part of the next-sentence prediction objective. Unlike positional embeddings (which have one vector per position up to the maximum sequence length) or token embeddings (which have one vector per vocabulary entry), segment embeddings are an extremely compact table: just two learned vectors that modulate the entire input based on segment membership.

For single-sentence inputs, all tokens receive type ID 0. The segment B embedding is irrelevant but still part of the model. For sentence-pair inputs, the type IDs switch from 0 to 1 at the second segment:

Modern models like RoBERTa dropped segment embeddings entirely. Since RoBERTa discarded next-sentence prediction during pre-training, there was no objective that rewarded the model for learning meaningful segment embeddings. Removing segment embeddings simplifies the architecture without hurting performance. Interestingly, even models that handle sentence pairs for tasks like Natural Language Inference work fine without explicit segment embeddings, relying instead on the [SEP] tokens and positional context to infer segment boundaries. This empirical finding suggests that [SEP]'s boundary signal may already be sufficient, and segment embeddings are redundant rather than necessary.

Let's trace through the complete construction of a BERT input for a sentence-pair task, step by step, to make the three-component embedding concrete. Suppose the task is Natural Language Inference, and the input is:

• Premise: "The cat sat on the mat."
• Hypothesis: "A cat was resting."

Step 1: WordPiece tokenization. The tokenizer first splits each sentence into subword tokens. "Resting" might become ["rest", "##ing"] if "resting" is not a vocabulary entry but "rest" and the suffix "##ing" are.

Step 2: Special token insertion. The tokenizer prepends [CLS] to the entire sequence, inserts [SEP] after sentence A, appends sentence B's tokens, and adds a final [SEP]:

Step 3: Embedding summation. For each position $i$, three vectors are looked up and summed: the token embedding for the token at position $i$, the positional embedding for position $i$, and the segment embedding for the type ID at position $i$ (either row 0 or row 1 of the segment embedding table).

Step 4: Transformer processing. The summed embeddings enter the first transformer layer. All 16 positions attend to each other (with full bidirectional attention). After all 12 layers, each position has a contextualized representation that encodes information from the entire sequence.

Step 5: Classification. The [CLS] representation at position 0 is passed through a classification head with three outputs: entailment, contradiction, neutral. The class with the highest softmax probability is the predicted label.

The segment boundary created by the [SEP] at position 8 and the jump in type IDs from 0 to 1 gives the model two complementary signals about where sentence A ends and sentence B begins. The attention mechanism can learn to use either signal: some heads may track the [SEP] boundary while others track the type ID boundary. Having both signals makes the boundary information resilient to noise in either one.

The figure below visualizes this three-layer structure: token identity, positional index, and segment membership all encoded in parallel for the same sequence.

Special tokens receive their own rows in the embedding matrix, just like regular vocabulary tokens. However, their embeddings have unusual properties compared to word tokens. Understanding these properties helps explain some otherwise puzzling behaviors of BERT-based models.

Unlike word tokens, which learn embeddings that encode linguistic and semantic content (semantic similarity, syntactic role, distributional context), special token embeddings encode structural roles. The [CLS] embedding starts as a random initialization and learns during fine-tuning to be a useful "summarization query." By the end of fine-tuning, the [CLS] embedding occupies a region of embedding space associated with "global summary of the input," shaped by whatever task was used for fine-tuning. Different fine-tuning tasks will pull the [CLS] embedding toward different regions, which is why [CLS] representations are not transferable across tasks without re-fine-tuning.

The [PAD] embedding is almost never receives a substantial update, since padding positions are masked out during attention and excluded from loss computation. Their gradients are effectively zero throughout training. As a result, the [PAD] embedding tends to remain close to its random initialization, occupying a somewhat arbitrary location in embedding space. This is not a problem: since [PAD] embeddings are never attended to (due to the attention mask) and never contribute to loss (due to the label mask), their location in embedding space doesn't affect model behavior.

The [MASK] embedding is interesting: it's used heavily during pre-training but never during fine-tuning or inference on real tasks. The [MASK] token appears at approximately 15% of positions in every pre-training batch, so its embedding receives large gradient signal during pre-training. The model must learn to look past [MASK] and use surrounding context instead of the masked token's embedding. Some researchers have noted that the [MASK] embedding occupies a distinct region of embedding space, clearly separated from all word token embeddings. This reflects its purely structural rather than linguistic role. This separation makes sense: the model has learned that [MASK] means "ignore my embedding identity and use context instead," which naturally pushes the [MASK] embedding away from any region occupied by real words.

When fine-tuning with adapter layers or parameter-efficient methods, the embeddings of special tokens are typically not frozen even when other token embeddings are. The fine-tuning objective often depends critically on specific special tokens (the [CLS] representation for classification) and their embeddings need to participate in gradient updates. Freezing the [CLS] embedding while fine-tuning the classification head would prevent the [CLS] embedding from adapting to the task, significantly degrading performance. This is a practical point worth remembering when designing parameter-efficient fine-tuning schemes.

Most tokenizer frameworks allow you to add custom special tokens beyond the defaults. This is useful when building task-specific models that require additional structural signals not present in the base tokenizer. Custom special tokens are one of the most powerful tools for adapting pre-trained models to novel tasks and domains without architectural changes.

The key distinction between a custom special token and a regular vocabulary token is how the tokenizer handles it. A regular multi-character string like "summarize" will be tokenized as one or more subword pieces based on the vocabulary: "sum", "##mar", "##ize" in WordPiece, for example. A custom special token, by contrast, is treated as an atomic unit that is never split, regardless of whether its character sequence appears in the base vocabulary. This atomicity is what makes special tokens reliable as structural signals: you know they will always appear as a single ID, never as a fragmented sequence.

The implications of atomicity are more important than they might first appear. Suppose you are building a document processing model and you want to use [SECTION_HEADER] as a structural marker. If this were treated as a regular string, it might tokenize as ["[", "section", "_", "header", "]"] or some other fragmented representation. The model would have no reliable way to recognize it as a boundary token because its representation would be constructed from content token embeddings that have no structural meaning. As a custom special token, it receives a single dedicated embedding row whose meaning the model learns entirely from how it appears in the training data: always at section boundaries.

Common use cases for custom special tokens include:

• Document structure tokens: [TITLE], [BODY], [FOOTER] for document processing models that need to distinguish different textual zones
• Domain markers: [SQL], [PYTHON], [MARKDOWN] for multi-domain code models that handle different programming languages
• Task tokens: <summarize>, <translate>, <question> for multi-task models where the same model handles different tasks conditioned on a token
• Entity markers: <PERSON>, <LOCATION>, <ORG> for named entity recognition pipelines that mark entity spans
• Control tokens: <formal>, <informal> for style-controlled generation where the desired output style is specified as a token

T5 uses this pattern extensively, though in a slightly different form. Its tokenizer prepends a task prefix to each input, such as "translate English to French: " or "summarize: ". In T5's implementation, these prefixes are treated as regular tokens rather than special tokens in the strict sense: they can be split by the tokenizer and their IDs come from the base vocabulary. The model still learns to condition on them through gradient updates, but they don't have the atomicity guarantee of true special tokens. This design choice reflects T5's text-to-text framing, where the goal is to minimize the distance between pre-training and fine-tuning by using natural text-like task descriptions rather than arbitrary structural tokens.

Chat-fine-tuned models use custom special tokens to enforce conversational structure. LLaMA 2 Chat uses [INST] and [/INST] to bracket user turns, and <<SYS>> and <</SYS>> for system prompts. Llama 3 uses <|begin_of_text|>, <|start_header_id|>, <|end_header_id|>, and <|eot_id|>. These tokens are not arbitrary: they were chosen to be unlikely in natural text, reducing the risk that a user-provided string could accidentally trigger the structural signals. This is a security consideration as much as a design one, since an attacker could attempt to inject turn-boundary tokens to manipulate the model's behavior. This type of attack, called prompt injection, exploits the fact that the model cannot distinguish user-provided text from model-provided structural tokens if the user knows the token strings. Some systems address this by using tokens whose character strings are unlikely to appear in user input and by enforcing that users cannot directly inject raw token IDs.

When adding custom special tokens, you must ensure they receive dedicated embedding rows and that the model's embedding layer is resized to accommodate them. This is a multi-step process that requires careful sequencing to avoid corrupting the model.

The four required steps are:

1. Adding the token to the tokenizer's special tokens list, so the tokenizer knows to treat it atomically
2. Resizing the model's embedding layer to cover the new vocabulary size, adding new rows to the embedding table
3. Initializing the new embedding rows (either randomly or with a better warm-start strategy)
4. Fine-tuning the model so the new token's embedding learns a useful representation through gradient updates

The initialization strategy for new token embeddings matters more than it might seem. Random initialization places the new token's embedding in an arbitrary location in embedding space. The model must then learn through gradient updates where the embedding should be, which requires sufficient fine-tuning data to converge. If you have limited fine-tuning data, random initialization can leave the new token's embedding in a poorly calibrated region, producing unstable or unpredictable model behavior.

A better strategy is to initialize new token embeddings as the mean of the embeddings of related tokens. If you are adding [ENGLISH], initializing its embedding as the average of the BERT embeddings for the tokens in the word "English" (that is, the embeddings for "english" in BERT-base-uncased) gives the model a warm start that is semantically close to the intended concept. The model still needs to learn the structural role of [ENGLISH] (which is different from the word "English" appearing in normal text), but the warm start reduces the fine-tuning distance. HuggingFace's resize_token_embeddings handles the table extension but uses random initialization by default, so you may want to manually override the new rows with mean-of-related-tokens initialization.

A second initialization strategy that works well in practice is to initialize the new token's embedding with the mean of all existing token embeddings. This places the new token at the centroid of the embedding space, which is semantically neutral but better than a random outlier. From the centroid, the model's gradient updates can move the embedding in whichever direction the task requires, with a shorter expected distance than from a random starting point.

Let's work through a complete implementation demonstrating special token behavior using the transformers library. The goal is to make every concept from the preceding sections concrete and verifiable.

We start by loading a BERT tokenizer and model to inspect how special tokens are handled:

Every special token has a reserved ID in the vocabulary. Inspecting these IDs confirms the conventions discussed in the theory sections:

The [PAD] token receives ID 0, which is a common convention allowing zero-initialized padding tensors. The other tokens occupy specific reserved slots at the start of the BERT vocabulary. [CLS] (101), [SEP] (102), and [MASK] (103) are assigned consecutive IDs. These IDs mark them as core structural tokens.

Now let's see how special tokens are automatically inserted during tokenization. The add_special_tokens=True parameter (the default) instructs the tokenizer to automatically wrap the input with the appropriate structural tokens:

The [CLS] token appears at position 0, [SEP] at the end, and all tokens receive type ID 0 and attention mask value 1, since there is no padding. The automatic insertion of [CLS] and [SEP] is handled entirely by the tokenizer: you pass a plain string and receive a properly framed sequence. This automation is convenient but can hide the special tokens from view, which is why it's worth examining the output explicitly at least once to understand what the tokenizer is doing.

For tasks like Natural Language Inference, BERT takes two sentences as input. The tokenizer handles the two-sentence format automatically, inserting the correct special tokens and generating the type IDs:

Notice how sentence A tokens receive type ID 0 and sentence B tokens receive type ID 1. The second [SEP] token, which closes sentence B, also receives type ID 1. The [CLS] token and the first [SEP] both receive type ID 0, since they are part of the "sentence A" region. This is the standard BERT convention: [CLS] and the first [SEP] are attributed to sentence A, while sentence B's closing [SEP] is attributed to sentence B.

Padding becomes visible when we process sequences of different lengths together. This demonstrates why the attention mask is required for correctness:

Let's reproduce BERT's masking strategy to see exactly how [MASK] tokens are introduced during pre-training. Understanding this procedure at the code level reinforces the 80/10/10 design described in the theory section:

Let's add domain-marker tokens to signal the input language for a multilingual scenario. This demonstrates the full workflow of extending a tokenizer and resizing the embedding layer:

The custom tokens are treated as atomic units: they will not be split by the WordPiece algorithm, and their IDs point to new rows in the embedding table that can be trained during fine-tuning. The resize_token_embeddings call extends the model's input embedding matrix and output projection layer to accommodate the new vocabulary entries. These new rows are randomly initialized by default, so fine-tuning on sufficient data is required for them to learn meaningful representations.

The key parameters for special token handling in HuggingFace are:

• add_special_tokens: When set to True in tokenizer(...), automatically inserts [CLS], [SEP], and other required tokens. Defaults to True. Set to False only when you need manual control over special token placement.
• return_token_type_ids: When set to True, returns the segment ID tensor alongside input_ids. Required for BERT sentence-pair tasks.
• return_attention_mask: When set to True, returns the attention mask tensor. Defaults to True in most contexts. Always inspect this tensor when debugging unexpected model behavior.
• padding: Controls how sequences in a batch are padded. True pads to the longest sequence in the batch; "max_length" pads to a fixed length specified by max_length.
• truncation: When set to True with max_length, sequences exceeding the limit are cut to fit. For sentence pairs, truncation applies to the longer of the two sentences by default.

Autoregressive generation with models like GPT-2 or LLaMA involves different special token mechanics than BERT's bidirectional encoding. Understanding these differences is necessary for building reliable generation pipelines, and confusing the two paradigms is a frequent source of bugs.

In autoregressive generation, the <BOS> token is typically prepended to the prompt before the model begins generating. The model then predicts one token at a time, appending each prediction to the input and re-running the forward pass (or using cached key-value pairs for efficiency). The absence of a [CLS]-like classification token reflects the fundamental difference in how these models are used: BERT reads a complete sequence and produces a representation; GPT reads a growing prefix and produces the next token. There is no equivalent to [CLS] in autoregressive models because there is no single aggregate representation of the whole sequence at inference time.

Generation terminates when one of three conditions is met:

1. The model produces the <EOS> token, indicating that the sequence is naturally complete
2. A maximum sequence length is reached, preventing runaway generation
3. A stop sequence specified by the caller is detected, allowing task-specific termination

The first condition requires the most judgment. The model learns to produce <EOS> when it has completed a natural unit of text by seeing <EOS> at the end of every training document. At inference time, the model's learned sense of "completeness" determines when it outputs <EOS>. This can be unreliable for out-of-distribution prompts where the model has no clear sense of when it should stop. The maximum length limit (condition 2) is a safety valve for this case.

A common pitfall is double-<BOS> injection: if you manually prepend <BOS> to your prompt and also use tokenizer(text, add_special_tokens=True), many tokenizers will add another <BOS>. This corrupts the input in subtle ways because the model has never seen double-<BOS> in training and its behavior at the first generated token will be unpredictable. Always check whether your tokenizer is handling special token insertion or whether you're handling it manually, and never do both.

A second pitfall specific to chat models is the incorrect placement of turn-boundary tokens. Each chat-fine-tuned model has a specific template for how conversation turns are formatted, and deviating from this template produces degraded responses. A model fine-tuned to expect [INST] ... [/INST] will not perform correctly if the user's message is provided without these markers, because the model has learned strong associations between response quality and the presence of the correct structural tokens. Many HuggingFace models now include a chat_template in their tokenizer configuration that automatically applies the correct formatting when you call tokenizer.apply_chat_template(messages).

For encoder-decoder models like T5 and BART, the decoder always begins generation from a decoder-start token (often </s> for T5 or <s> for BART). The encoder processes the full input with end-of-sequence markers, while the decoder generates from its start token, cross-attending to the encoder's output. The two sides of the encoder-decoder architecture use different special token conventions because they serve different roles: the encoder reads a complete input (so it needs boundary markers), while the decoder generates step by step (so it needs a start signal and will produce an end signal when done).

Special tokens also interact with stopping criteria in non-obvious ways. If the model is generating a multi-turn conversation and you want it to stop at the assistant's turn boundary, you need to ensure the turn-separator token is in the stopping criteria. This is model-specific, since different chat-fine-tuned models use different tokens (<|im_end|>, [/INST], <|eot_id|>, and others) to mark the end of assistant turns. Failing to include the correct stop token causes the model to continue generating past the natural stopping point, potentially producing responses that bleed into the next conversation turn.

Special tokens elegantly solve the problem of communicating structure to a model without architectural changes, but they carry several limitations worth understanding. These limitations are not obscure edge cases: they affect every practitioner who deploys transformer-based models in production.

The pre-training/fine-tuning mismatch for [MASK] is the most documented issue. The token that is central to BERT's pre-training objective is entirely absent during downstream use. The fine-tuned model is never exposed to [MASK] except during pre-training, yet the pre-training process sees [MASK] at 12% of positions on average (15% selected, of which 80% become [MASK]). This asymmetry means the model's representation of non-masked positions during pre-training is subtly different from its representation during fine-tuning: in pre-training, the model "knows" it might need to reconstruct any position (since any position could be masked in a different batch), while in fine-tuning, all positions are real. This is why models like SpanBERT (which replaces contiguous spans) and XLNet (which uses permutation language modeling without [MASK]) achieve better downstream performance on many tasks: they reduce or eliminate the pre-training/fine-tuning gap.

Padding introduces a practical inefficiency that becomes severe with variable-length text. In a batch where sequences vary widely in length (which is common with real-world text), a large fraction of positions may be padding. This wasted computation scales quadratically with padding length in attention (every real token attends to all positions, including padded ones, before masking). Techniques like "dynamic padding" (padding only to the longest sequence in each batch rather than a global maximum) and "packing" (concatenating multiple short examples into a single long sequence, separated by [SEP] tokens) address this. Packing requires careful attention masking to prevent examples from attending to each other across the [SEP] boundary, which adds implementation complexity. The reward is substantial: packing can reduce training compute by 50% or more when the average sequence length is much shorter than the maximum.

The [CLS] representation for sentence embeddings is often misunderstood, and this misunderstanding has real consequences. Many practitioners use raw BERT's [CLS] output for semantic similarity tasks without fine-tuning. This approach consistently underperforms because BERT's pre-training objective (next-sentence prediction) does not train [CLS] for general semantic equivalence. The NSP task asks a coarse yes/no question about sentence adjacency, not a graded semantic similarity question. Models like Sentence-BERT explicitly fine-tune with a contrastive sentence similarity objective, using labeled sentence pairs, to produce high-quality [CLS]-based embeddings. This fine-tuning step is not optional if you want reliable semantic search. The original Sentence-BERT paper showed that raw BERT [CLS] embeddings performed worse than averaging random GloVe embeddings for semantic textual similarity, a result that should calibrate expectations about what pre-training alone provides.

Custom special tokens are useful but require careful management across the model lifecycle. If you add a [ENGLISH] token and save the tokenizer, you must also save the resized model. This ensures the embedding rows for the new tokens are included. Loading the base pre-trained model without the resize will silently produce embedding lookups for out-of-range IDs, which in many deep learning frameworks wraps around or triggers undefined behavior. The model and tokenizer must always be saved and loaded as a matched pair when custom tokens have been added.

The diversity of special token conventions across model families creates friction when combining components from different ecosystems. A BERT tokenizer's [SEP] and a T5 tokenizer's </s> play similar structural roles but have different IDs, different embedding spaces, and different positional conventions. When porting a fine-tuned head from one base model to another, the special token handling must be explicitly reconfigured, and the fine-tuned head may need to be retrained because the [CLS]-equivalent representation from the new model carries different information. These cross-model compatibility issues are a recurring practical challenge in production NLP systems that mix components from different pre-trained checkpoints.

Finally, custom special tokens in chat models introduce a security surface that is easy to overlook. If a model uses [INST] to mark user turns, a user who knows this convention could attempt to inject [/INST] into their input to prematurely close their turn and inject artificial assistant text. Defense against this attack typically involves either sanitizing user inputs to remove special token strings or using token sequences that are unlikely to appear in normal text. Some systems run the user input through the tokenizer and check for unexpected special token IDs before passing the input to the model. This is not a hypothetical concern: prompt injection exploiting turn-boundary tokens has been demonstrated in practice on deployed chat systems.

Special tokens are the control structures of the tokenizer interface. They inject structural information into plain token ID sequences, enabling a single model architecture to handle classification, sequence labeling, generation, and sentence-pair reasoning by changing only how inputs are framed.

The key tokens and their roles are:

• [CLS]: Provides an aggregate input representation for classification tasks; its final hidden state feeds into classification heads after fine-tuning. Use mean pooling instead of raw [CLS] when working with pre-trained models without task-specific fine-tuning.
• [SEP]: Marks sentence or segment boundaries; combined with token type IDs, distinguishes segments in multi-sentence inputs. Even single-sentence inputs require a closing [SEP].
• [PAD]: Fills shorter sequences to enable batching; must be masked in attention computation and excluded from loss calculation using the standard -100 ignore index.
• [MASK]: Enables BERT's MLM pre-training by marking positions the model must predict; the 80/10/10 replacement strategy reduces the training/inference mismatch by ensuring non-[MASK] tokens appear at masked positions during pre-training.
• [UNK]: Handles out-of-vocabulary characters or subwords; rarely seen in well-designed tokenizers with character-level coverage or byte fallback, but worth monitoring as a distribution shift indicator in production systems.

Beyond BERT's vocabulary, <BOS> and <EOS> tokens serve analogous roles in generative models, framing generation start and completion. Token type IDs provide segment-level structural signals via learned segment embeddings, though modern models like RoBERTa have shown these can be dropped without performance loss.

The most important practical guidelines when working with special tokens are: always use a model's corresponding tokenizer to ensure ID assignments are consistent; verify that padding is masked in attention and excluded from loss computation; when adding custom special tokens, resize the model's embedding layer before fine-tuning and consider mean-of-related-tokens initialization; and when working with sentence embeddings, fine-tune [CLS] for your specific similarity task rather than relying on its raw pre-trained state.

The next chapter examines the practical challenges that arise when tokenizers encounter difficult inputs: numbers, code, multilingual text, emoji, and adversarial inputs designed to exploit tokenization artifacts.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about special tokens in transformer models.