Instruction Format › Multi-Turn Conversation Format

Real conversations involve multiple exchanges between the user and the assistant. A single question and answer might suffice for simple tasks, but complex interactions require context to build over time. A multi-turn format must track the full conversation history while clearly marking each turn's boundaries and roles. This creates a cumulative context where each response can draw on everything that came before. This makes possible the model to resolve pronouns, follow up on previous statements, and maintain a coherent narrative across many exchanges.

Understanding multi-turn format is necessary because it determines how the model encodes conversational memory. The model does not have explicit memory; it has context. All prior turns are re-encoded on every generation step, and the model's responses are conditioned on the entire visible history. The format's job is to make that history unambiguously parseable so the model can correctly identify which content came from which speaker at each turn.

A simple multi-turn structure looks like:

The model processes the entire history when generating each new response. For the second question, it sees both the original question about France's capital, the assistant's response, and the follow-up question about population. This context allows it to resolve the ambiguous reference "What's the population?" to mean Paris's population. Without the conversation history, the model would have no way to know what "the population" refers to. Multi-turn formats let natural, contextual conversation instead of isolated question-answering.

Multi-turn training requires careful attention to what receives gradient updates. Typically, you only compute loss on the assistant turns, not user turns or the system message. Standard causal attention masking ensures only assistant tokens contribute to the loss, preventing the model from learning to predict user inputs. User turns give context that shapes the assistant's response, but they are not targets for prediction. This selective loss computation is what makes the model specialized for response generation rather than conversation modeling in the general sense.

The training loss objective for a multi-turn example can be written as follows. Let $\mathbf{x} = (x_1, x_2, \ldots, x_T)$ denote the full token sequence including system, user, and assistant tokens, and let $\mathcal{A} \subset \{1, \ldots, T\}$ denote the set of positions corresponding to assistant response tokens. The instruction-tuning loss is:

$$\mathcal{L} = -\sum_{t \in \mathcal{A}} \log P(x_t \mid x_1, \ldots, x_{t-1}; \theta)$$

where:

• $x_t$ is the token at position $t$
• $x_1, \ldots, x_{t-1}$ is the full causal context up to position $t$
• $\theta$ are the model parameters
• $\mathcal{A}$ is the set of assistant-turn token positions

The key insight is that $\mathcal{A}$ does not include all positions, only those belonging to assistant turns. Non-assistant positions are still present in the input and influence the attention computation through the conditioning context $x_1, \ldots, x_{t-1}$, but they do not contribute to the gradient signal. This means the model learns to generate good assistant responses while freely using system and user context to condition those responses.

The conversation format must handle several edge cases that arise in real-world applications. What happens if a user message is empty? What if there are consecutive user messages without assistant responses? How should the model handle extremely long conversation histories that exceed context length? Different formats and training procedures handle these cases differently, and your choice depends on your application's requirements. Some applications might truncate long histories, while others might summarize earlier turns, and still others might refuse to continue conversations that exceed certain lengths. None of these strategies is universally correct; each involves tradeoffs between context richness and computational feasibility.