Instruction Format › Tokenizing for Training
Language AI HandbookAll chapters Instruction Format › Tokenizing for TrainingOne section · Sections in this chapter8 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
During instruction tuning, you need to format the conversation and create labels that mask out non-assistant tokens from the loss calculation. This loss masking is basic to instruction tuning because it focuses the learning signal on the intended target: teaching the model to generate good responses. Without loss masking, the model would spend half its capacity learning to predict user messages, system prompts, and formatting tokens, diluting the gradient signal for what matters.
Loss masking works by replacing the labels for non-assistant tokens with -100, a value that PyTorch's CrossEntropyLoss function treats as "ignore this position." The loss at each position is computed as the cross-entropy between the model's predicted probability distribution and the true next token. When the label is -100, that position contributes zero to the total loss and zero to the gradient. Only positions where the label is a valid token ID contribute to learning.
Let's see how to prepare training data end to end:
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
# Ensure we have a pad token
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
# Training conversation with complete assistant response
training_messages = [
{"role": "system", "content": "You are a math tutor."},
{"role": "user", "content": "What is 15 + 27?"},
{"role": "assistant", "content": "15 + 27 = 42"},
]
# Tokenize the full conversation
import torch
def _to_tensor(ids):
"""Convert apply_chat_template output to a 2D tensor regardless of version."""
if isinstance(ids, torch.Tensor):
return ids.unsqueeze(0) if ids.dim() == 1 else ids
# BatchEncoding (dict-like) or list
if hasattr(ids, "input_ids"):
ids = ids.input_ids
if isinstance(ids, torch.Tensor):
return ids.unsqueeze(0) if ids.dim() == 1 else ids
# plain Python list of ints
flat = (
ids[0]
if (isinstance(ids, list) and ids and isinstance(ids[0], list))
else ids
)
return torch.tensor([flat])
_full_ids = tokenizer.apply_chat_template(
training_messages,
tokenize=True,
add_generation_prompt=False, # Don't add prompt since we have the response
)
full_tokens = _to_tensor(_full_ids)Full sequence length: 35 tokens Full sequence tokens: <|system|> You are a math tutor.<|end|><|user|> What is 15 + 27?<|end|><|assistant|> 15 + 27 = 42<|end|><|endoftext|>
The decoded sequence shows the conversation history merged into a single string with appropriate control tokens. Notice that we set add_generation_prompt=False because this is training data with a complete assistant response already included. We do not need to prompt for generation; we need to present the complete example for the model to learn from. If we accidentally set it to True, we would append the start-of-assistant marker a second time, corrupting the token sequence.
For training, we need to identify which tokens are from the assistant response, so we only compute loss on those:
# Get everything up to the assistant response
messages_without_response = training_messages[:-1]
_prompt_ids = tokenizer.apply_chat_template(
messages_without_response,
tokenize=True,
add_generation_prompt=True, # Add assistant turn start
)
prompt_tokens = _to_tensor(_prompt_ids)
# The assistant response tokens are the difference
prompt_length = prompt_tokens.shape[1]
full_length = full_tokens.shape[1]
response_length = full_length - prompt_lengthPrompt length: 22 tokens Response length: 13 tokens Prompt: <|system|> You are a math tutor.<|end|><|user|> What is 15 + 27?<|end|><|assistant|> Response tokens only: 15 + 27 = 42<|end|><|endoftext|>
These outputs confirm the separation between the instruction context and the target response. By tokenizing the conversation with and without the assistant response, we can precisely identify which tokens constitute the response. This identification is important because those are the only tokens that should contribute to the training loss. The technique of comparing two tokenizations is called the prompt-continuation split, and it is the most reliable way to find the boundary between context and generation target.
Now we can create labels with masking for the prompt tokens:
# Create labels for training
# -100 is the ignore index for CrossEntropyLoss
labels = full_tokens.clone()
labels[0, :prompt_length] = -100 # Mask out prompt tokens from lossToken-by-token label visualization: ------------------------------------------------------------ 0: '<|system|>' -> masked 1: 'You' -> masked 2: 'are' -> masked 3: 'a' -> masked 4: 'math' -> masked 5: 't' -> masked 6: 'utor' -> masked 7: '.' -> masked 8: '<|end|>' -> masked 9: '<|user|>' -> masked 10: 'What' -> masked 11: 'is' -> masked 12: '' -> masked 13: '1' -> masked 14: '5' -> masked 15: '+' -> masked 16: '' -> masked 17: '2' -> masked 18: '7' -> masked 19: '?' -> masked 20: '<|end|>' -> masked 21: '<|assistant|>' -> masked 22: '' -> loss 23: '1' -> loss 24: '5' -> loss 25: '+' -> loss 26: '' -> loss 27: '2' -> loss 28: '7' -> loss 29: '=' -> loss 30: '' -> loss 31: '4' -> loss 32: '2' -> loss 33: '<|end|>' -> loss 34: '<|endoftext|>' -> loss
The tokens marked as "loss" are from the assistant response and contribute to training. The masked tokens are from the system message, the user message, and format tokens. During backpropagation, gradients only flow from the assistant tokens, teaching the model to generate appropriate responses while preserving its understanding of how to parse the format structure. The special value -100 is recognized by PyTorch's CrossEntropyLoss as an ignore index, causing those positions to be excluded from the loss calculation entirely. Any attempt to include padding tokens in the loss would corrupt training with meaningless gradient signal, so the same masking technique also applies to padding positions when batching variable-length sequences.
The Loss Fraction Problem
An important but often overlooked issue with loss masking is what practitioners call the loss fraction problem: in a heavily masked training example, only a small fraction of tokens contribute to the loss. If a training example has 200 context tokens and 20 response tokens, 90% of the tokens are masked. The model processes the entire 220-token sequence but only receives gradient updates from 20 positions.
This creates two related problems. First, short responses are implicitly downweighted relative to long responses: a dataset where some examples have 5-token responses and others have 500-token responses will train more effectively on the long-response examples by a factor of 100 in terms of gradient signal per example. Second, conversations with very short responses waste most of the compute spent processing the context.
Common solutions include filtering the training dataset to ensure responses meet a minimum length, upweighting examples with high response-to-context ratios, normalizing the loss by the number of non-masked tokens per example rather than per sequence, and using sequence packing to maximize response tokens per batch. The normalization strategy is particularly important: a naive implementation that averages loss over all token positions will systematically undervalue short, dense-information responses compared to long, verbose ones.