Special Tokens › Custom Special Tokens

Most tokenizer frameworks allow you to add custom special tokens beyond the defaults. This is useful when building task-specific models that require additional structural signals not present in the base tokenizer. Custom special tokens are one of the most powerful tools for adapting pre-trained models to novel tasks and domains without architectural changes.

The key distinction between a custom special token and a regular vocabulary token is how the tokenizer handles it. A regular multi-character string like "summarize" will be tokenized as one or more subword pieces based on the vocabulary: "sum", "##mar", "##ize" in WordPiece, for example. A custom special token, by contrast, is treated as an atomic unit that is never split, regardless of whether its character sequence appears in the base vocabulary. This atomicity is what makes special tokens reliable as structural signals: you know they will always appear as a single ID, never as a fragmented sequence.

The implications of atomicity are more important than they might first appear. Suppose you are building a document processing model and you want to use [SECTION_HEADER] as a structural marker. If this were treated as a regular string, it might tokenize as ["[", "section", "_", "header", "]"] or some other fragmented representation. The model would have no reliable way to recognize it as a boundary token because its representation would be constructed from content token embeddings that have no structural meaning. As a custom special token, it receives a single dedicated embedding row whose meaning the model learns entirely from how it appears in the training data: always at section boundaries.

Common use cases for custom special tokens include:

• Document structure tokens: [TITLE], [BODY], [FOOTER] for document processing models that need to distinguish different textual zones
• Domain markers: [SQL], [PYTHON], [MARKDOWN] for multi-domain code models that handle different programming languages
• Task tokens: <summarize>, <translate>, <question> for multi-task models where the same model handles different tasks conditioned on a token
• Entity markers: <PERSON>, <LOCATION>, <ORG> for named entity recognition pipelines that mark entity spans
• Control tokens: <formal>, <informal> for style-controlled generation where the desired output style is specified as a token

T5 uses this pattern extensively, though in a slightly different form. Its tokenizer prepends a task prefix to each input, such as "translate English to French: " or "summarize: ". In T5's implementation, these prefixes are treated as regular tokens rather than special tokens in the strict sense: they can be split by the tokenizer and their IDs come from the base vocabulary. The model still learns to condition on them through gradient updates, but they don't have the atomicity guarantee of true special tokens. This design choice reflects T5's text-to-text framing, where the goal is to minimize the distance between pre-training and fine-tuning by using natural text-like task descriptions rather than arbitrary structural tokens.

Chat-fine-tuned models use custom special tokens to enforce conversational structure. LLaMA 2 Chat uses [INST] and [/INST] to bracket user turns, and <<SYS>> and <</SYS>> for system prompts. Llama 3 uses <|begin_of_text|>, <|start_header_id|>, <|end_header_id|>, and <|eot_id|>. These tokens are not arbitrary: they were chosen to be unlikely in natural text, reducing the risk that a user-provided string could accidentally trigger the structural signals. This is a security consideration as much as a design one, since an attacker could attempt to inject turn-boundary tokens to manipulate the model's behavior. This type of attack, called prompt injection, exploits the fact that the model cannot distinguish user-provided text from model-provided structural tokens if the user knows the token strings. Some systems address this by using tokens whose character strings are unlikely to appear in user input and by enforcing that users cannot directly inject raw token IDs.

When adding custom special tokens, you must ensure they receive dedicated embedding rows and that the model's embedding layer is resized to accommodate them. This is a multi-step process that requires careful sequencing to avoid corrupting the model.

The four required steps are:

1. Adding the token to the tokenizer's special tokens list, so the tokenizer knows to treat it atomically
2. Resizing the model's embedding layer to cover the new vocabulary size, adding new rows to the embedding table
3. Initializing the new embedding rows (either randomly or with a better warm-start strategy)
4. Fine-tuning the model so the new token's embedding learns a useful representation through gradient updates

The initialization strategy for new token embeddings matters more than it might seem. Random initialization places the new token's embedding in an arbitrary location in embedding space. The model must then learn through gradient updates where the embedding should be, which requires sufficient fine-tuning data to converge. If you have limited fine-tuning data, random initialization can leave the new token's embedding in a poorly calibrated region, producing unstable or unpredictable model behavior.

A better strategy is to initialize new token embeddings as the mean of the embeddings of related tokens. If you are adding [ENGLISH], initializing its embedding as the average of the BERT embeddings for the tokens in the word "English" (that is, the embeddings for "english" in BERT-base-uncased) gives the model a warm start that is semantically close to the intended concept. The model still needs to learn the structural role of [ENGLISH] (which is different from the word "English" appearing in normal text), but the warm start reduces the fine-tuning distance. HuggingFace's resize_token_embeddings handles the table extension but uses random initialization by default, so you may want to manually override the new rows with mean-of-related-tokens initialization.

A second initialization strategy that works well in practice is to initialize the new token's embedding with the mean of all existing token embeddings. This places the new token at the centroid of the embedding space, which is semantically neutral but better than a random outlier. From the centroid, the model's gradient updates can move the embedding in whichever direction the task requires, with a shorter expected distance than from a random starting point.