Instruction Format › Chat Templates

Chat templates standardize how conversations are converted to token sequences. Standardization ensures the training and inference formats match. Different model families use different templates. This shows different design choices and historical developments. Using the wrong template for a given model produces poor results because the model encounters structural cues it has never seen before. Understanding the major template formats helps you work effectively with the open-source model ecosystem and make informed decisions when designing your own instruction-tuning pipelines.

Think of chat templates as dialects of a common conversational language. All dialects convey the same basic information (who said what, when), but they do so using different vocabulary and syntax. A speaker of one dialect can be understood only by listeners who know that dialect. Similarly, a model trained on ChatML can be "understood" only by inference code that speaks ChatML.

ChatML, which stands for Chat Markup Language, was introduced by OpenAI and has become one of the most common formats in the open-source ecosystem. It uses special tokens to mark message boundaries, giving clear and unambiguous structure:

The key elements are designed for clarity and consistency:

• <|im_start|> marks the beginning of a message
• <|im_end|> marks the end of a message
• The role immediately follows <|im_start|> on the same line
• Message content follows the role on subsequent lines

ChatML's structure makes role boundaries unambiguous. The special tokens are unlikely to appear in normal text, and the consistent structure simplifies parsing both for the model and for any code that needs to process these conversations. Many open-source models, including various Qwen versions, use ChatML or close variants. The widespread adoption of ChatML makes it easier to share training data and inference code across different projects.

The name "im" in <|im_start|> stands for "imaginary monologue," which shows the original design intent that each message is one participant's contribution to a dialogue. This naming is a historical artifact; in practice, everyone uses it simply as a structural marker. The single-token nature of <|im_start|> and <|im_end|> is important: the tokenizer maps each entire string to a single vocabulary entry, meaning the model always processes the boundary signal with a single attention step rather than needing to integrate information across multiple subword tokens.

Meta's Llama models use a different format that evolved across model generations. This shows lessons learned and changing design priorities. Llama 2's chat format uses:

The format nests the system message within special <<SYS>> tags, places instructions within [INST] and [/INST] markers, and uses <s> and </s> as sequence boundaries. For multi-turn conversations, each turn is wrapped with these markers. This nesting structure shows a hierarchical view of the conversation where system content is embedded within the first instruction turn rather than standing alone. Critics of this approach noted that it makes the system message structurally different from how it is conceptually: it is supposed to be persistent and separate, but it appears as a sub-component of the first user turn.

Llama 3 simplified and modified this format materially, introducing new special tokens like <|begin_of_text|>, <|start_header_id|>, and <|end_header_id|>:

This format gives more explicit structure with dedicated tokens for headers, end-of-turn markers, and text boundaries. The evolution from Llama 2 to Llama 3 format shows how template design improves as practitioners gain experience. The Llama 3 format separates concerns more cleanly: there are distinct tokens for marking role headers versus marking content boundaries, making the structure more regular and easier to parse programmatically. The system message now stands on equal footing with user and assistant turns structurally, which better shows its conceptual role.

Mistral models use a simpler format that minimizes token overhead:

Mistral's format combines the system message with the first user message and uses minimal special tokens. This design choice prioritizes simplicity and efficiency. Notice there is no space after [/INST] before the assistant response. This detail matters for exact token alignment. During training, the model learns that the assistant response begins immediately after the closing instruction marker. Introducing a space at inference time would create a token sequence the model has never encountered, potentially affecting output quality. The Mistral format is unforgiving of whitespace errors in a way that ChatML is not, because ChatML uses newlines as natural separators.

Mistral's decision to embed the system message in the first user turn is a pragmatic simplification. It reduces the number of distinct roles the model must learn to handle, at the cost of making the system message structurally indistinguishable from the first user query. This is a reasonable tradeoff for models deployed in contexts where system messages are predictable and stable, but it limits fine-grained behavioral control in complex deployments.

Zephyr models, built on the Mistral architecture but fine-tuned for alignment, use a ChatML-like format with different tokens:

The structure resembles ChatML but uses <|system|>, <|user|>, and <|assistant|> as role markers with </s> as the end-of-message delimiter. This hybrid approach combines the clarity of ChatML-style role markers with the familiarity of the standard end-of-sequence token. The choice of format shows the Zephyr team's design preferences and the training data they used, which was sourced largely from UltraChat and other datasets that used this structure. The use of </s> as the end-of-message token rather than a dedicated <|im_end|> means the format shares structural tokens with the model's natural language generation, which can occasionally cause the model to end turns prematurely if the end-of-sequence token appears in normal content.

Microsoft's Phi-3 family uses a format that blends elements of both ChatML and Llama 3:

The <|end|> token marks message boundaries, while <|system|>, <|user|>, and <|assistant|> mark role starts. The format is clean and easy to parse, with role markers on their own lines and content following on the next line. Phi-3 models were designed to reach strong performance with small parameter counts, and the clean template structure contributes to efficient training signal extraction. The regular pattern of "role marker, newline, content, end marker" is easy for the model to internalize and reliable to variations in content length.