Autoregressive Generation › Stopping Criteria
Language AI HandbookAll chapters Autoregressive Generation › Stopping CriteriaOne section · Sections in this chapter5 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Generation must stop at some point. Without explicit stopping criteria, the model would continue generating tokens indefinitely, or until hitting a maximum length set by the system. Several strategies determine when to halt generation, and choosing the right one matters both for output quality and for computational efficiency. Stopping too early truncates incomplete thoughts; stopping too late wastes compute and may produce degenerate repetition.
The choice of stopping criterion is tightly coupled to the task. A question-answering system wants to stop when the answer is complete, not when some fixed token count is reached. A chat application wants to stop at the end of the assistant's turn, before starting the next user turn. A code generation tool might want to stop after a complete function definition. In each case, the optimal stopping criterion encodes task-specific knowledge about what "done" means, and generic length limits are a blunt instrument compared to structured stop conditions.
It is worth understanding how stop tokens are introduced during training. Models are trained on documents that naturally contain EOS tokens at their ends. When the training objective is to predict the next token given all previous ones, the model learns to assign high probability to the EOS token when it has "said everything there is to say" about a topic. This learned behavior transfers to generation: a well-trained model will produce EOS at the right moment naturally, which makes it the most semantically aware stopping mechanism available.
End-of-Sequence Token
The most common stopping criterion is the end-of-sequence (EOS) token. During training, sequences end with a special token that signals completion. When the model generates this token, we stop:
def generate_until_eos(model, tokenizer, prompt, max_length=100):
"""Generate until EOS token or max length."""
input_ids = tokenizer.encode(prompt, return_tensors="pt")
for _ in range(max_length):
with torch.no_grad():
outputs = model(input_ids)
next_token_id = outputs.logits[:, -1, :].argmax(
dim=-1, keepdim=True
)
input_ids = torch.cat([input_ids, next_token_id], dim=1)
# Stop if EOS generated
if next_token_id.item() == tokenizer.eos_token_id:
break
return tokenizer.decode(input_ids[0], skip_special_tokens=True)The EOS token works well for tasks with natural endings, like completing a sentence or answering a question. However, some prompts don't have clear endpoints, and the model may never generate EOS. Instruction-tuned models are trained to use EOS reliably because their fine-tuning data includes explicit turn boundaries, but base language models trained purely on raw text may rarely emit EOS in open-ended generation. For base models, a length limit is often necessary as a safety net.
Maximum Length
A hard maximum length prevents runaway generation:
def generate_with_max_length(model, tokenizer, prompt, max_new_tokens=50):
"""Generate exactly max_new_tokens (unless EOS encountered)."""
input_ids = tokenizer.encode(prompt, return_tensors="pt")
prompt_length = input_ids.size(1)
max_total_length = prompt_length + max_new_tokens
while input_ids.size(1) < max_total_length:
with torch.no_grad():
outputs = model(input_ids)
next_token_id = outputs.logits[:, -1, :].argmax(
dim=-1, keepdim=True
)
input_ids = torch.cat([input_ids, next_token_id], dim=1)
if next_token_id.item() == tokenizer.eos_token_id:
break
return tokenizer.decode(input_ids[0], skip_special_tokens=True)Stop Sequences
For specific applications, we might want to stop when certain text patterns appear. This is common in chat applications where we stop at user turn markers:
def generate_until_stop_sequence(
model,
tokenizer,
prompt: str,
stop_sequences: list[str],
max_new_tokens: int = 100,
):
"""Generate until a stop sequence is found in the output."""
input_ids = tokenizer.encode(prompt, return_tensors="pt")
for _ in range(max_new_tokens):
with torch.no_grad():
outputs = model(input_ids)
next_token_id = outputs.logits[:, -1, :].argmax(
dim=-1, keepdim=True
)
input_ids = torch.cat([input_ids, next_token_id], dim=1)
# Check for stop sequences in generated text
current_text = tokenizer.decode(input_ids[0], skip_special_tokens=True)
for stop_seq in stop_sequences:
if stop_seq in current_text[len(prompt) :]:
# Truncate at stop sequence
stop_idx = current_text.find(stop_seq, len(prompt))
return current_text[:stop_idx]
if next_token_id.item() == tokenizer.eos_token_id:
break
return tokenizer.decode(input_ids[0], skip_special_tokens=True)Prompt: List three colors: 1. Red 2. Stop sequences: ['\n\n', '4.'] Generated: List three colors: 1. Red 2. Blue 3. Yellow
The generation stopped as soon as it encountered one of our specified patterns. This is particularly useful for structured outputs where you know the format in advance, such as generating exactly three items in a list or stopping at the end of an assistant's turn in a conversation.
One practical implementation challenge with stop sequences is that they may span multiple tokens. The string "\n\n" (double newline) might tokenize to a single token in some tokenizers or to two separate newline tokens in others. The check must operate on the decoded text string rather than on token IDs, which means decoding the generated tokens at each step. This adds some overhead compared to a simple token-ID comparison, but it is the only reliable way to detect multi-token stop sequences. For performance-necessary applications, it is worth pre-tokenizing common stop sequences and checking token IDs directly when possible.
Multiple Stopping Criteria
In practice, we often combine multiple criteria. The Hugging Face library supports this through StoppingCriteria objects:
from transformers import StoppingCriteria
class MaxTokensCriteria(StoppingCriteria):
"""Stop after generating max_tokens new tokens."""
def __init__(self, start_length: int, max_new_tokens: int):
self.start_length = start_length
self.max_new_tokens = max_new_tokens
def __call__(self, input_ids, scores, **kwargs):
return input_ids.shape[1] >= self.start_length + self.max_new_tokens
class StopStringCriteria(StoppingCriteria):
"""Stop when a specific string appears in the output."""
def __init__(self, tokenizer, stop_string: str, start_length: int):
self.tokenizer = tokenizer
self.stop_string = stop_string
self.start_length = start_length
def __call__(self, input_ids, scores, **kwargs):
generated_text = self.tokenizer.decode(
input_ids[0, self.start_length :], skip_special_tokens=True
)
return self.stop_string in generated_text