Instruction Format › Format Considerations for Training Data

The format you choose affects several aspects of instruction tuning, from computational efficiency to model quality. Understanding these tradeoffs helps you make principled decisions when designing your training pipeline rather than simply defaulting to whatever the most popular current model uses.

Token efficiency matters for training speed and cost. Some formats use verbose markers that consume many tokens per message. A format that adds 20 tokens of overhead per turn might seem negligible, but across millions of training examples and dozens of turns per conversation, that overhead becomes substantial. ChatML's <|im_start|> and <|im_end|> are single tokens in models trained with them, but if you are adding them to a base model that has never seen them, they might be tokenized into multiple pieces initially. Compare the token counts of different formats on your data to understand the overhead and make informed decisions about format selection.

Consistency with pre-training improves results. If you are fine-tuning a model that was pre-trained with a specific format, continuing to use that format builds on what the model already learned. The model has already developed internal representations for those format tokens and patterns. This is why using the official chat template for instruct models is important: they were trained expecting that exact format, and any deviation forces the model to generalize in ways it was not prepared for. Even small inconsistencies, like using a slightly different whitespace pattern, can cause measurable degradation.

Edge cases need explicit handling. What happens when a message is empty? When the user sends multiple messages before the assistant responds? When the conversation exceeds context length and must be truncated? Your template and data processing pipeline should handle these cases gracefully. Consider whether empty messages should be included with empty content or skipped entirely. Define clear truncation strategies that preserve the most important context, typically the system message and the most recent exchanges.

System messages are not always supported. Some formats and models do not support system messages, or support them only in limited ways. Mistral's original chat format had limited system message support, requiring the system content to be prepended to the first user message. If your application relies on system prompts for behavior control, verify your chosen model and format support them properly before committing to a particular approach. Deploying a system-message-dependent application on a model that was trained without dedicated system message support will result in unreliable adherence to system instructions.