Instruction Format › Working with Chat Templates in Practice
Language AI HandbookAll chapters Instruction Format › Working with Chat Templates in PracticeOne section · Sections in this chapter7 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The Hugging Face transformers library gives a standardized way to apply chat templates through the tokenizer's apply_chat_template method. This ensures you use the correct format for each model without manually constructing the template strings, which would be error-prone and tedious. The library abstracts away the differences between formats, letting you to write code that works across multiple model families.
The abstraction layer that apply_chat_template gives is more useful than it might first appear. Without it, you would need to maintain model-specific formatting code for every model you work with, update that code whenever a model's template changes, and carefully test that your code produces bit-identical output to the model's training pipeline. With it, you can write a single piece of code that formats conversations correctly for any model that has a template registered in its tokenizer. The templates are stored as Jinja2 strings in the tokenizer's tokenizer_config.json file, so they travel with the model weights.
Let's explore how chat templates work with a concrete example:
from transformers import AutoTokenizer
# Load a tokenizer that has a chat template defined
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
# Define a conversation as a list of message dictionaries
messages = [
{"role": "system", "content": "You are a helpful coding assistant."},
{
"role": "user",
"content": "Write a Python function to calculate factorial.",
},
{
"role": "assistant",
"content": "def factorial(n):\n if n <= 1:\n return 1\n return n * factorial(n - 1)",
},
{"role": "user", "content": "Can you add input validation?"},
]The conversation is represented as a list of dictionaries, each with a role and content key. This standardized format works across different models regardless of their underlying template structure. The tokenizer handles converting it to the model-specific template, so you can write model-agnostic code that processes conversations uniformly. The dictionary representation is also the natural format for APIs like OpenAI's Chat Completions endpoint, so this same conversation structure often flows from API response logging directly into training data preparation.
# Apply the chat template to get the formatted string
formatted_prompt = tokenizer.apply_chat_template(
messages,
tokenize=False, # Return string, not token IDs
add_generation_prompt=True, # Add the assistant turn start
)Formatted conversation:
<|system|>
You are a helpful coding assistant.<|end|>
<|user|>
Write a Python function to calculate factorial.<|end|>
<|assistant|>
def factorial(n):
if n <= 1:
return 1
return n * factorial(n - 1)<|end|>
<|user|>
Can you add input validation?<|end|>
<|assistant|>The output displays the conversation formatted with the model's control tokens. The add_generation_prompt=True parameter is important for inference. It adds whatever tokens signal the start of an assistant turn, so the model knows to generate a response. Without it, you would get the conversation history but no prompt for the model to continue. The model would see a complete conversation and have no indication that it should add anything. This is one of the most common bugs in inference pipelines: forgetting add_generation_prompt=True and then wondering why the model either refuses to generate or generates continuation text rather than a fresh response.
Let's examine how the same conversation looks with different model templates:
# Compare templates across different models
model_names = [
"microsoft/Phi-3-mini-4k-instruct",
"Qwen/Qwen2-0.5B-Instruct",
"mistralai/Mistral-7B-Instruct-v0.1",
]
formatted_outputs = {}
for model_name in model_names:
tok = AutoTokenizer.from_pretrained(model_name)
# Use a simpler conversation for comparison
simple_messages = [
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi there!"},
{"role": "user", "content": "How are you?"},
]
formatted = tok.apply_chat_template(
simple_messages, tokenize=False, add_generation_prompt=True
)
formatted_outputs[model_name] = formatted============================================================ Model: microsoft/Phi-3-mini-4k-instruct ============================================================ <|user|> Hello!<|end|> <|assistant|> Hi there!<|end|> <|user|> How are you?<|end|> <|assistant|> ============================================================ Model: Qwen/Qwen2-0.5B-Instruct ============================================================ <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user Hello!<|im_end|> <|im_start|>assistant Hi there!<|im_end|> <|im_start|>user How are you?<|im_end|> <|im_start|>assistant ============================================================ Model: mistralai/Mistral-7B-Instruct-v0.1 ============================================================ <s> [INST] Hello! [/INST] Hi there!</s> [INST] How are you? [/INST]
Notice how each model uses different delimiters and structures. The Phi model uses its specific template markers. Qwen uses ChatML-style <|im_start|> and <|im_end|> tokens. Mistral uses [INST] and [/INST] markers. Using the wrong template for a model would produce nonsensical outputs because the model has only been trained to recognize its specific format. This comparison illustrates why the apply_chat_template abstraction is so useful: it shields you from needing to know these details for every model you work with. When you load a model and call apply_chat_template, you are guaranteed to get exactly the format that model was trained on.
