Special Tokens › Handling Special Tokens in Generation

Autoregressive generation with models like GPT-2 or LLaMA involves different special token mechanics than BERT's bidirectional encoding. Understanding these differences is necessary for building reliable generation pipelines, and confusing the two paradigms is a frequent source of bugs.

In autoregressive generation, the <BOS> token is typically prepended to the prompt before the model begins generating. The model then predicts one token at a time, appending each prediction to the input and re-running the forward pass (or using cached key-value pairs for efficiency). The absence of a [CLS]-like classification token reflects the fundamental difference in how these models are used: BERT reads a complete sequence and produces a representation; GPT reads a growing prefix and produces the next token. There is no equivalent to [CLS] in autoregressive models because there is no single aggregate representation of the whole sequence at inference time.

Generation terminates when one of three conditions is met:

1. The model produces the <EOS> token, indicating that the sequence is naturally complete
2. A maximum sequence length is reached, preventing runaway generation
3. A stop sequence specified by the caller is detected, allowing task-specific termination

The first condition requires the most judgment. The model learns to produce <EOS> when it has completed a natural unit of text by seeing <EOS> at the end of every training document. At inference time, the model's learned sense of "completeness" determines when it outputs <EOS>. This can be unreliable for out-of-distribution prompts where the model has no clear sense of when it should stop. The maximum length limit (condition 2) is a safety valve for this case.

A common pitfall is double-<BOS> injection: if you manually prepend <BOS> to your prompt and also use tokenizer(text, add_special_tokens=True), many tokenizers will add another <BOS>. This corrupts the input in subtle ways because the model has never seen double-<BOS> in training and its behavior at the first generated token will be unpredictable. Always check whether your tokenizer is handling special token insertion or whether you're handling it manually, and never do both.

A second pitfall specific to chat models is the incorrect placement of turn-boundary tokens. Each chat-fine-tuned model has a specific template for how conversation turns are formatted, and deviating from this template produces degraded responses. A model fine-tuned to expect [INST] ... [/INST] will not perform correctly if the user's message is provided without these markers, because the model has learned strong associations between response quality and the presence of the correct structural tokens. Many HuggingFace models now include a chat_template in their tokenizer configuration that automatically applies the correct formatting when you call tokenizer.apply_chat_template(messages).

For encoder-decoder models like T5 and BART, the decoder always begins generation from a decoder-start token (often </s> for T5 or <s> for BART). The encoder processes the full input with end-of-sequence markers, while the decoder generates from its start token, cross-attending to the encoder's output. The two sides of the encoder-decoder architecture use different special token conventions because they serve different roles: the encoder reads a complete input (so it needs boundary markers), while the decoder generates step by step (so it needs a start signal and will produce an end signal when done).

Special tokens also interact with stopping criteria in non-obvious ways. If the model is generating a multi-turn conversation and you want it to stop at the assistant's turn boundary, you need to ensure the turn-separator token is in the stopping criteria. This is model-specific, since different chat-fine-tuned models use different tokens (<|im_end|>, [/INST], <|eot_id|>, and others) to mark the end of assistant turns. Failing to include the correct stop token causes the model to continue generating past the natural stopping point, potentially producing responses that bleed into the next conversation turn.