Special Tokens › Beginning and End of Sequence Tokens

Not all models use BERT's [CLS]/[SEP] convention. GPT-style autoregressive models, which generate text left-to-right, use different markers for sequence boundaries. The difference in convention reflects a fundamental difference in how these models process text: BERT reads the entire sequence at once and produces contextual representations for every position, while GPT-style models read left-to-right and produce each output token from the context of all preceding tokens.

The shift from encoder-style to decoder-style models brought a corresponding shift in how sequence boundaries are marked. BERT needs to know where each sentence is because it processes the whole sequence simultaneously: the [CLS] and [SEP] tokens provide spatial landmarks in an already-complete sequence. GPT-style models, by contrast, process sequences one token at a time in order: what they need to know is when to start generating and when to stop. This difference in need produces a difference in token design.

Autoregressive models like GPT use beginning-of-sequence (<BOS>) and end-of-sequence (<EOS>) tokens to mark the start and completion of generated text. These tokens play a role similar to [CLS] and [SEP] in BERT, but the analogy is imperfect: <BOS> and <EOS> are about temporal boundaries in a generation process, while [CLS] and [SEP] are about spatial boundaries in a structured input.

The <BOS> token (sometimes written <s> or <|startoftext|>) primes the model for generation. At inference time, generation begins from a prompt that may or may not include this token explicitly. When present, it signals to the model that what follows is the beginning of a new document or generation context. The model's first prediction is conditioned on everything up to and including <BOS>. During training, every sequence in the training corpus is prepended with <BOS>, so the model learns to associate this token with "the beginning of coherent text."

The <EOS> token (sometimes </s> or <|endoftext|>) signals that generation should terminate. During training, sequences end with <EOS>, teaching the model to predict this token when a sequence is naturally complete. At inference time, generation loops until <EOS> is produced (or a maximum length limit is reached). The key insight is that the model learns during training that <EOS> follows the natural conclusion of a text: a story that has reached its end, an answer that is complete, a list that has been fully enumerated. At inference time, the model applies this learned association to decide when to stop, which is why some models generate coherent text that stops cleanly while others trail off or repeat themselves.

GPT-2 uses a single token <|endoftext|> for both purposes, prepending it to documents during training to distinguish document boundaries in long concatenated training sequences. During generation, the model learns that producing this token signals the end of a coherent piece of text. This dual role (marking both the start of one document and the end of the previous one) is a practical simplification that reflects GPT-2's training on a single concatenated stream of text rather than discrete batched sequences.

Encoder-decoder models like T5 use </s> as both a sentence separator (replacing BERT's [SEP]) and an end-of-sequence marker. This reflects T5's text-to-text framing where all tasks are cast as sequence generation. In T5, the encoder processes the input text ending with </s>, and the decoder generates the output text, also ending with </s>. The same token serves both roles because the architecture treats input and output symmetrically as text strings.

Special tokens are assigned specific IDs in the vocabulary, typically reserved at predictable positions. For example, in the BERT tokenizer:

• [PAD] is assigned ID 0 (allowing zero-initialization of padding tensors, since the zero vector is a natural default)
• [UNK] receives ID 100
• [CLS] receives ID 101
• [SEP] receives ID 102
• [MASK] receives ID 103

These conventions are baked into model configurations and are not interchangeable. When loading a pre-trained model, always use its corresponding tokenizer. Mismatched ID assignments are a common source of silent failures where the model silently processes the wrong signals: if you use a different tokenizer that assigns [CLS] to ID 200, the model will look up embedding row 200 when it expects row 101, producing nonsense embeddings for the most important structural token.

A related pitfall is loading a tokenizer without the corresponding special token configuration. HuggingFace's AutoTokenizer.from_pretrained() handles this correctly, loading both the vocabulary and the special token assignments together. Manual vocabulary construction from raw files does not preserve these assignments unless the configuration files are also loaded.