Special Tokens › Code Implementation
Language AI HandbookAll chapters Special Tokens › Code ImplementationOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's work through a complete implementation demonstrating special token behavior using the transformers library. The goal is to make every concept from the preceding sections concrete and verifiable.
Setup and Imports
We start by loading a BERT tokenizer and model to inspect how special tokens are handled:
from transformers import BertTokenizer
# Load BERT base tokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")Examining Special Token IDs
Every special token has a reserved ID in the vocabulary. Inspecting these IDs confirms the conventions discussed in the theory sections:
# Inspect special token IDs
special_token_info = {
"[CLS]": tokenizer.cls_token_id,
"[SEP]": tokenizer.sep_token_id,
"[PAD]": tokenizer.pad_token_id,
"[MASK]": tokenizer.mask_token_id,
"[UNK]": tokenizer.unk_token_id,
}Special Token IDs: [CLS] -> ID 101 [SEP] -> ID 102 [PAD] -> ID 0 [MASK] -> ID 103 [UNK] -> ID 100
The [PAD] token receives ID 0, which is a common convention allowing zero-initialized padding tensors. The other tokens occupy specific reserved slots at the start of the BERT vocabulary. [CLS] (101), [SEP] (102), and [MASK] (103) are assigned consecutive IDs. These IDs mark them as core structural tokens.
Single Sentence Tokenization
Now let's see how special tokens are automatically inserted during tokenization. The add_special_tokens=True parameter (the default) instructs the tokenizer to automatically wrap the input with the appropriate structural tokens:
sentence = "The cat sat on the mat."
# Tokenize with special tokens
encoded = tokenizer(sentence, return_tensors="pt", return_token_type_ids=True)
# Decode each token ID back to its string
token_ids = encoded["input_ids"][0].tolist()
tokens = tokenizer.convert_ids_to_tokens(token_ids)
token_types = encoded["token_type_ids"][0].tolist()
attention_mask = encoded["attention_mask"][0].tolist()Single sentence tokenization: Token ID Type Mask ------------------------------ [CLS] 101 0 1 the 1996 0 1 cat 4937 0 1 sat 2938 0 1 on 2006 0 1 the 1996 0 1 mat 13523 0 1 . 1012 0 1 [SEP] 102 0 1
The [CLS] token appears at position 0, [SEP] at the end, and all tokens receive type ID 0 and attention mask value 1, since there is no padding. The automatic insertion of [CLS] and [SEP] is handled entirely by the tokenizer: you pass a plain string and receive a properly framed sequence. This automation is convenient but can hide the special tokens from view, which is why it's worth examining the output explicitly at least once to understand what the tokenizer is doing.
Sentence Pair Tokenization
For tasks like Natural Language Inference, BERT takes two sentences as input. The tokenizer handles the two-sentence format automatically, inserting the correct special tokens and generating the type IDs:
sentence_a = "The cat sat on the mat."
sentence_b = "A cat was resting."
# Tokenize sentence pair
encoded_pair = tokenizer(
sentence_a,
sentence_b,
return_tensors="pt",
return_token_type_ids=True,
padding=True,
)
pair_ids = encoded_pair["input_ids"][0].tolist()
pair_tokens = tokenizer.convert_ids_to_tokens(pair_ids)
pair_types = encoded_pair["token_type_ids"][0].tolist()
pair_mask = encoded_pair["attention_mask"][0].tolist()Sentence pair tokenization: Token ID Type Mask ------------------------------ [CLS] 101 0 1 the 1996 0 1 cat 4937 0 1 sat 2938 0 1 on 2006 0 1 the 1996 0 1 mat 13523 0 1 . 1012 0 1 [SEP] 102 0 1 a 1037 1 1 cat 4937 1 1 was 2001 1 1 resting 8345 1 1 . 1012 1 1 [SEP] 102 1 1
Notice how sentence A tokens receive type ID 0 and sentence B tokens receive type ID 1. The second [SEP] token, which closes sentence B, also receives type ID 1. The [CLS] token and the first [SEP] both receive type ID 0, since they are part of the "sentence A" region. This is the standard BERT convention: [CLS] and the first [SEP] are attributed to sentence A, while sentence B's closing [SEP] is attributed to sentence B.
Padding in Batches
Padding becomes visible when we process sequences of different lengths together. This demonstrates why the attention mask is required for correctness:
sentences = [
"Short sentence.",
"This is a slightly longer sentence with more words.",
"A medium length sentence here.",
]
batch = tokenizer(
sentences,
return_tensors="pt",
padding=True,
truncation=True,
max_length=20,
)
batch_tokens = [
tokenizer.convert_ids_to_tokens(ids.tolist()) for ids in batch["input_ids"]
]Padded batch (showing token IDs and attention masks): Sentence 1: Tokens: [CLS] short sentence . [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] Mask: 1 1 1 1 1 0 0 0 0 0 0 0 Sentence 2: Tokens: [CLS] this is a slightly longer sentence with more words . [SEP] Mask: 1 1 1 1 1 1 1 1 1 1 1 1 Sentence 3: Tokens: [CLS] a medium length sentence here . [SEP] [PAD] [PAD] [PAD] [PAD] Mask: 1 1 1 1 1 1 1 1 0 0 0 0
Shorter sequences are padded with [PAD] tokens to match the length of the longest sequence in the batch. The attention mask marks these positions with 0, and the model will exclude them from its attention computation. The padding ensures all sequences in the batch have the same tensor dimensions, enabling GPU matrix operations. Without the attention mask, the padding tokens would contaminate the real token representations.
The heatmap below shows the attention mask for the padded batch, making the active versus masked positions immediately visible:
The MLM Masking Procedure
Let's reproduce BERT's masking strategy to see exactly how [MASK] tokens are introduced during pre-training. Understanding this procedure at the code level reinforces the 80/10/10 design described in the theory section:
def apply_mlm_masking(token_ids, tokenizer, mask_prob=0.15, seed=42):
"""Apply BERT-style MLM masking: 80% [MASK], 10% random, 10% unchanged."""
rng = np.random.default_rng(seed)
labels = [-100] * len(token_ids) # -100 means "ignore this position"
masked_ids = token_ids.copy()
# Special token positions that should never be masked
special_ids = {
tokenizer.cls_token_id,
tokenizer.sep_token_id,
tokenizer.pad_token_id,
}
for i, token_id in enumerate(token_ids):
if token_id in special_ids:
continue # Never mask special tokens
if rng.random() < mask_prob:
labels[i] = token_id # Store original for loss computation
r = rng.random()
if r < 0.80:
masked_ids[i] = tokenizer.mask_token_id # Replace with [MASK]
elif r < 0.90:
masked_ids[i] = int(
rng.integers(100, len(tokenizer))
) # Random token
# else: keep original (remaining 10%)
return masked_ids, labels
sentence = "The quick brown fox jumps over the lazy dog."
token_ids = tokenizer.encode(sentence)
tokens_before = tokenizer.convert_ids_to_tokens(token_ids)
masked_ids, labels = apply_mlm_masking(token_ids, tokenizer, seed=7)
tokens_after = tokenizer.convert_ids_to_tokens(masked_ids)MLM masking demonstration:
Position Before After Label
------------------------------------------------
0 [CLS] [CLS] -
1 the the -
2 quick quick -
3 brown brown -
4 fox fox -
5 jumps jumps -
6 over over -
7 the max the <-- masked
8 lazy lazy -
9 dog dog -
10 . . -
11 [SEP] [SEP] -The masking procedure shows the 80/10/10 split in action: some positions receive the explicit [MASK] token, some are replaced with a random vocabulary token, and some remain unchanged. In all three cases, the model is trained to predict the original token at the label position (stored as labels[i]). Positions with label = -100 are excluded from the loss computation: the model is only evaluated on positions that were selected for masking, not on all positions simultaneously.
The special token positions ([CLS] and [SEP]) are explicitly excluded from masking. This exclusion is important for two reasons. First, the model needs reliable structural tokens: if [CLS] were sometimes replaced with a random token, the model could not reliably use the [CLS] position for classification. Second, the model should not need to predict [CLS] and [SEP] from context, since they carry structural rather than semantic information and their correct placement is determined by the tokenizer, not the text content.
To see the 80/10/10 distribution at scale, we can apply the masking strategy over many events and count how the selected positions break down:
![Bar chart showing proportions of [MASK], random, and unchanged token replacements in MLM masking.](https://assets.mbrenndoerfer.com/_optimized/notebooks/special_tokens_files/mlm-masking-strategy-distribution-dark-1920w.webp)
Adding Custom Special Tokens
Let's add domain-marker tokens to signal the input language for a multilingual scenario. This demonstrates the full workflow of extending a tokenizer and resizing the embedding layer:
from transformers import BertModel
# Add a custom special token for domain marking
custom_tokens = ["[ENGLISH]", "[FRENCH]", "[SPANISH]"]
num_added = tokenizer.add_special_tokens(
{"additional_special_tokens": custom_tokens}
)
# Resize the model embedding layer
model = BertModel.from_pretrained("bert-base-uncased")
model.resize_token_embeddings(len(tokenizer))
# Verify the new tokens got IDs
new_token_ids = {
tok: tokenizer.convert_tokens_to_ids(tok) for tok in custom_tokens
}Added 3 new special tokens to vocabulary New vocabulary size: 30525 Custom special token IDs: [ENGLISH] -> ID 30522 [FRENCH] -> ID 30523 [SPANISH] -> ID 30524 Tokenized '[ENGLISH] The cat sat on the mat.': ['[ENGLISH]', 'the', 'cat', 'sat', 'on', 'the', 'mat', '.']
The custom tokens are treated as atomic units: they will not be split by the WordPiece algorithm, and their IDs point to new rows in the embedding table that can be trained during fine-tuning. The resize_token_embeddings call extends the model's input embedding matrix and output projection layer to accommodate the new vocabulary entries. These new rows are randomly initialized by default, so fine-tuning on sufficient data is required for them to learn meaningful representations.
Key Parameters Summary
The key parameters for special token handling in HuggingFace are:
add_special_tokens: When set toTrueintokenizer(...), automatically inserts[CLS],[SEP], and other required tokens. Defaults toTrue. Set toFalseonly when you need manual control over special token placement.return_token_type_ids: When set toTrue, returns the segment ID tensor alongsideinput_ids. Required for BERT sentence-pair tasks.return_attention_mask: When set toTrue, returns the attention mask tensor. Defaults toTruein most contexts. Always inspect this tensor when debugging unexpected model behavior.padding: Controls how sequences in a batch are padded.Truepads to the longest sequence in the batch;"max_length"pads to a fixed length specified bymax_length.truncation: When set toTruewithmax_length, sequences exceeding the limit are cut to fit. For sentence pairs, truncation applies to the longer of the two sentences by default.