RLHF Pipeline › Stage 1: Supervised Fine-Tuning
Language AI HandbookAll chapters RLHF Pipeline › Stage 1: Supervised Fine-TuningOne section · Sections in this chapter3 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Supervised Fine-Tuning turns a pretrained language model into one able to following instructions and engaging in dialogue. While the pretrained model has acquired extensive knowledge and language understanding, it lacks the ability to respond helpfully to your queries in a conversational format. SFT closes this gap by exposing the model to paired examples of instructions and ideal responses, teaching it both the expected interaction pattern and the quality bar it should aim for.
Think of SFT as teaching an incredibly knowledgeable but socially awkward expert how to communicate. Before SFT, the language model has read essentially all of human-written text and internalized an enormous amount of knowledge. But it communicates the way text on the internet communicates, sometimes as a forum post, sometimes as a Wikipedia article, sometimes as a fictional story, depending on what pattern the prompt activates. After SFT, it has learned that when a human poses a question, the appropriate response is a clear, structured, helpful answer, not a continuation of the question into a series of related questions, not a fictional dialogue where characters discuss the topic, and not a Wikipedia-style article about the broader subject area. The format is part of the skill.
The SFT stage is also where the model develops an initial grasp of what "helpful" looks like in concrete terms. Through exposure to carefully crafted demonstrations, it learns that a good response is appropriately scoped to the question, acknowledges limitations when relevant, structures complex information clearly, and avoids unnecessary caveats or padding. These are difficult qualities to specify algorithmically, but relatively easy for expert human writers to demonstrate. SFT exploits that asymmetry, using human demonstrations as a richer signal than any reward function could capture.
Why SFT Comes First
You might wonder why we don't skip directly to reinforcement learning. The reason is practical: PPO optimization requires a model that already produces reasonable responses. Trying to optimize a raw pretrained model with RL is like trying to teach someone chess strategy before they know how the pieces move.
The pretrained model generates plausible continuations of text, but it doesn't understand the assistant paradigm. Given a question, it might generate more questions, continue with a different topic, or produce text that reads like training data rather than a helpful response. SFT provides the foundation by:
- Teaching the instruction-following format (recognizing prompts, generating responses)
- Establishing a baseline quality level that RL can refine
- Reducing the search space for PPO (responses are already in a useful format)
There is also a subtler reason that SFT matters for reward model training. The preference data collected for stage 2 requires the SFT model to generate candidate responses. If those candidates are incoherent, the human annotators comparing them face an impossible task: they cannot distinguish useful quality differences between incoherent responses. High-quality SFT outputs ensure that the preference data captures differences in response quality rather than the contrast between merely incoherent and completely incoherent. The SFT model sets the quality floor for every subsequent stage.
SFT Data Requirements
SFT data consists of (prompt, response) pairs showing ideal assistant behavior. Quality matters far more than quantity. A few thousand high-quality demonstrations often outperform millions of lower-quality examples.
sft_examples = [
{
"prompt": "What causes rainbows to form?",
"response": """Rainbows form through a combination of refraction, reflection, and dispersion of sunlight in water droplets. Here's how it works:
1. **Refraction**: Sunlight enters a water droplet and bends (refracts) because light travels slower in water than in air.
2. **Dispersion**: Different wavelengths of light bend by different amounts, separating white light into its component colors (red bends least, violet bends most).
3. **Internal Reflection**: The light reflects off the back inner surface of the droplet.
4. **Second Refraction**: The light bends again as it exits the droplet, further separating the colors.
You see a rainbow when the sun is behind you and rain is in front of you, with the light which reflects back at approximately 42 degrees from the original light direction.""",
},
{
"prompt": "Write a short poem about coding",
"response": """Lines of Logic
In the glow of screens at night,
We craft our worlds in black and white,
Variables dance, functions call,
Building dreams that never fall.
Bugs may lurk in shadowed code,
Each one a puzzle to decode,
But when at last the program runs,
We've made magic, byte by byte, we've won.""",
},
]SFT Training Example: Prompt: What causes rainbows to form? Response preview: Rainbows form through a combination of refraction, reflection, and dispersion of sunlight in water droplets. Here's how it works: 1. **Refraction**: Sunlight enters a water droplet and bends (refract...
The demonstrations should exhibit properties you want the final model to have: helpfulness, appropriate tone, factual accuracy, and safety awareness. As discussed in the Instruction Data Creation chapter, these examples can come from human writers, filtered model outputs, or synthetic generation with quality controls.
What makes a good SFT dataset goes beyond having correct answers. The demonstrations need to model the right relationship between question complexity and response depth. A simple factual question deserves a concise answer; a complex reasoning question deserves a step-by-step explanation. Demonstrations that always use the same format, regardless of question type, teach the model the wrong thing: they teach it to mimic a style rather than to calibrate the response to the question. Building this calibration into the demonstration data is one of the craft elements that separates effective SFT datasets from mediocre ones.
Coverage breadth is equally important. The SFT model's capabilities are strongly constrained by the distribution of topics and task types in the training data. A model trained only on question-answering demonstrations will struggle with creative writing requests. A model trained only on English examples will struggle with multilingual queries. Systematic gaps in the SFT data create systematic gaps in the model's behavior that downstream RL training can partially compensate for but never fully overcome. This is why organizations building production alignment pipelines invest heavily in dataset curation: so diversity across domains, styles, difficulty levels, and instruction types.
SFT Training Process
SFT uses standard causal language modeling loss but only on the response tokens. This distinction is important for understanding how the model learns during this stage. The prompt provides context, letting the model to understand what kind of response is expected, but we don't penalize the model for not predicting prompt tokens. The reasoning is straightforward: we only want the model to learn how to respond given a prompt, not to memorize and reproduce the prompts themselves.
The training objective focuses exclusively on maximizing the likelihood of generating the correct response tokens, conditioned on the full context of the prompt. This targeted learning ensures the model develops the skill of creating appropriate responses rather than simply learning to continue arbitrary text.
import torch
from torch.utils.data import Dataset
class SFTDataset(Dataset):
"""Dataset for supervised fine-tuning with response-only loss."""
def __init__(self, examples, tokenizer, max_length=512):
self.examples = examples
self.tokenizer = tokenizer
self.max_length = max_length
def __len__(self):
return len(self.examples)
def __getitem__(self, idx):
example = self.examples[idx]
prompt_tokens = self.tokenizer.encode(example["prompt"])
response_tokens = self.tokenizer.encode(example["response"])
full_tokens = prompt_tokens + response_tokens
if len(full_tokens) > self.max_length:
full_tokens = full_tokens[: self.max_length]
response_start = len(prompt_tokens)
else:
response_start = len(prompt_tokens)
labels = [-100] * response_start + full_tokens[response_start:]
padding_length = self.max_length - len(full_tokens)
input_ids = full_tokens + [self.tokenizer.pad_token_id] * padding_length
labels = labels + [-100] * padding_length
return {
"input_ids": torch.tensor(input_ids),
"labels": torch.tensor(labels),
"attention_mask": torch.tensor(
[1] * len(full_tokens) + [0] * padding_length
),
}The key detail is setting labels to -100 for prompt tokens. This value is a special sentinel in PyTorch's CrossEntropyLoss function: any token position with a label of -100 is completely ignored during loss computation. By masking the prompt tokens this way, we ensure we only compute loss on the response portion, teaching the model to generate good responses without wasting gradient updates on predicting prompt content it doesn't need to reproduce.
def sft_training_step(model, batch, optimizer):
"""Single SFT training step with response-only loss."""
model.train()
input_ids = batch["input_ids"]
labels = batch["labels"]
attention_mask = batch["attention_mask"]
outputs = model(
input_ids=input_ids,
attention_mask=attention_mask,
labels=labels, # Model computes loss internally, ignoring -100 labels
)
loss = outputs.loss
optimizer.zero_grad()
loss.backward()
optimizer.step()
return loss.item()Key Parameters
SFT training is relatively straightforward compared to the later stages. The key parameters are:
- Learning rate: Typically 1e-5 to 5e-5 for full fine-tuning, or 1e-4 to 3e-4 for LoRA (as covered in the LoRA chapters)
- Batch size: 32 to 128 examples, depending on available memory
- Epochs: 1-3 passes over the data; more risks overfitting
- Warmup: 3-10% of total steps with linear warmup
sft_config = {
"learning_rate": 2e-5,
"batch_size": 64,
"num_epochs": 2,
"warmup_ratio": 0.03,
"weight_decay": 0.01,
"max_grad_norm": 1.0,
"lr_scheduler": "cosine",
}Typical SFT Configuration: learning_rate: 2e-05 batch_size: 64 num_epochs: 2 warmup_ratio: 0.03 weight_decay: 0.01 max_grad_norm: 1.0 lr_scheduler: cosine
Overfitting is a real concern with small SFT datasets. Monitor validation loss and stop training when it begins to increase. You might prefer training for slightly fewer steps than optimal to preserve model generalization.
The learning rate choice deserves particular attention. SFT is fine-tuning a model that has already learned enormous amounts from pretraining. Using too large a learning rate risks catastrophically forgetting pretraining knowledge, a problem where the model loses general capabilities in exchange for fitting the specific demonstrations. Using too small a learning rate means the model barely moves from its pretrained behavior, failing to learn the instruction-following format effectively. The 1e-5 to 5e-5 range for full fine-tuning represents decades of empirical wisdom about this trade-off. For LoRA fine-tuning, the larger learning rates (1e-4 to 3e-4) are appropriate because LoRA only updates a small fraction of parameters, so the effective learning rate on the full model is much smaller than the nominal value.
The number of epochs also warrants thought beyond "stop when validation loss increases." With SFT, the concern is overfitting in the traditional sense and sycophancy: the model learning to mimic the surface features of the demonstrations (bullet points, specific phrasing, characteristic openers) rather than the underlying quality. If you train for many epochs on a small dataset, the model may learn to reproduce specific demonstration responses almost verbatim rather than generalizing the quality signal. One to three epochs is the typical operating range precisely because it provides enough exposure to learn the patterns without enough repetition to memorize specific examples.
What Good SFT Looks Like
A well-trained SFT model should exhibit several distinguishing characteristics when you evaluate it qualitatively. When given a straightforward question, it should provide a direct and accurate answer without excessive preamble. When given a complex multi-step problem, it should structure its response with clear organization rather than presenting a wall of text. When given a request that falls outside appropriate assistant behavior, it should decline gracefully rather than either complying blindly or refusing with an unhelpful non-response.
The key insight is that SFT teaches the model a mode of operation rather than a set of facts. You can check whether SFT is working by observing whether the model consistently adopts the assistant persona across diverse prompt types. If the model responds to some prompts helpfully but reverts to raw language model behavior on others (generating continuations, creating multiple responses, or adopting the voice of a different persona), SFT has not been thorough enough. This inconsistency is usually a sign that either the demonstration data lacks coverage for certain prompt types or training ran for too few steps.