Function Calling: Structured Tool Use for Large Language Models › Function Calling Fine-Tuning

While general-purpose models like GPT-4 come with reliable function calling capabilities, open-source models often require fine-tuning to reliably emit structured tool calls. This process involves curating datasets of tool-use conversations and optimizing the model for schema adherence. Fine-tuning turns a base model from a general text generator into a specialized tool-calling agent.

The need for fine-tuning is about more than format compliance. A base model might generate syntactically valid JSON that is semantically wrong: it might call the right function but with hallucinated parameter values, or invoke a tool when the answer is already in its parametric knowledge. Fine-tuning teaches the model the output format and the judgment of when and how to invoke tools correctly. This judgment is fundamentally a classification problem: for every query, the model must decide among (a) answer directly, (b) request clarification, (c) call one or more tools.

Fine-tuning data for function calling follows a conversation format where each example consists of:

1. System Prompt: Defines available functions and general behavior guidelines
2. User Messages: Queries that require tool invocation
3. Assistant Messages: Either function calls (in the structured format) or final answers
4. Tool Results: Observations returned from executed functions
5. Final Answers: Natural language responses incorporating tool outputs

A single training example might look like this in ShareGPT format:

{
  "conversations": [
    {"from": "system", "value": "You are a helpful assistant with access to tools..."},
    {"from": "human", "value": "What's the weather in Paris?"},
    {"from": "gpt", "value": "<tool_call>{\"name\": \"get_weather\", \"arguments\": {\"location\": \"Paris\"}}</tool_call>"},
    {"from": "tool", "value": "{\"temperature\": 20, \"condition\": \"Sunny\"}"},
    {"from": "gpt", "value": "The weather in Paris is sunny with a temperature of 20 degrees Celsius."}
  ]
}

The necessary aspect is maintaining the exact output format expected at inference time. If the model is expected to wrap tool calls in <tool_call> tags, the training data must include these tags consistently. Inconsistency between training and inference formats leads to parsing errors and failed tool invocations. Additionally, training datasets should include diverse examples covering edge cases: tool calls with nested parameters, parallel invocations, error recoveries where the model must retry with corrected parameters, and clarifications where the model asks for missing required fields.

Negative examples are equally important. The dataset should include instances where the model should not call a tool, instead answering from its parametric knowledge. This prevents over-reliance on external tools and reduces unnecessary API calls and latency.

The diversity of the dataset matters as much as its size. A dataset with 10,000 examples covering only simple single-tool queries will produce a model that fails on multi-tool queries, ambiguous queries, and error recovery scenarios. A well-constructed dataset of 2,000 examples spanning all behavioral categories will generalize better. Think of the fine-tuning data as a specification of the system's behavioral contract: every pattern you want the model to handle correctly must appear somewhere in the training data.

Constructing high-quality fine-tuning data is expensive because it requires human annotation of correct tool-use behavior. Some teams have found success with synthetic data generation: using a powerful model (like GPT-4) to generate diverse queries and then annotating the correct tool calls programmatically, then using a weaker model to filter out low-quality examples. This approach can produce large, diverse datasets at reasonable cost but requires careful quality control to prevent the fine-tuned model from inheriting the generator model's failure modes.

The fine-tuning process uses standard supervised learning with cross-entropy loss, but with attention to special tokens that demarcate tool boundaries. As discussed in Instruction Tuning Training, the learning rate is typically lower than pre-training to preserve general capabilities while adapting to the specific tool-calling format. The model must learn the format and the semantics of when to transition between natural language and structured calls.

Key hyperparameters for function calling fine-tuning include:

• Learning rate: Often 1e-5 to 2e-5, smaller than general fine-tuning to maintain stability. Higher rates might cause catastrophic forgetting of conversational abilities or overfitting to specific tool formats.
• Sequence length: Must accommodate long tool descriptions and multi-turn conversations. Function schemas can be verbose, especially with nested objects, requiring context windows of 4k tokens or more.
• Masking strategy: Typically, only assistant messages (including tool calls) are used for loss computation, while user and system prompts are masked. This focuses the model's learning on generating appropriate responses rather than predicting your inputs.
• Data mixture: Combining tool-use data with general instruction data prevents catastrophic forgetting of conversational abilities, as explored in Catastrophic Forgetting. A typical mixture might be 70% tool-use data and 30% general instruction following data, though this varies by base model and target application.

The masking strategy deserves a closer look. When computing the cross-entropy loss, we want the model to learn to generate correct tool calls and final responses. We do not want it to expend capacity predicting the user's messages or the system prompt. By masking these tokens out of the loss computation (setting their weights to zero), we ensure the gradient signal comes entirely from the model's own outputs. This is the same technique used in instruction tuning more broadly and is important for efficient learning.

The training process must also account for the observation phase. Some implementations include simulated tool results in the training data, while others train only on the call generation phase and handle observations as context during inference. The former approach creates a more complete understanding of the tool-use cycle but requires a dataset of realistic tool outputs. Including observations in training is generally preferred because it teaches the model how to synthesize tool results into natural language responses, which is the final step that makes function calling useful to users.

Different model families require different fine-tuning approaches based on their native chat templates and special token vocabularies.

ChatML format (used by many open-source models) structures tool calls as separate message types:

Llama-2/3 chat formats use specific header tokens to distinguish between tool planning and final responses, often requiring the model to generate thinking tokens or plan steps before emitting the final JSON. These formats may require the model to first output reasoning in special tags, then the tool call, creating a chain-of-thought style approach that improves accuracy on complex multi-step problems.

The fine-tuning objective trains the model to recognize when its internal knowledge is insufficient (triggering a tool call) versus when it can answer directly. This calibration requires diverse training examples including:

• Tool-required queries: Questions that cannot be answered without external data, such as current weather or stock prices
• Direct answer queries: Questions within the model's parametric knowledge to prevent over-reliance on tools, such as historical facts or general knowledge
• Clarification requests: Ambiguous queries where the model should ask for parameter specification rather than hallucinate values, training the model to recognize uncertainty and request your input

The boundary between these categories is not always sharp, which is precisely what makes calibration difficult. "What's the population of Tokyo?" can be answered from parametric knowledge with reasonable accuracy (the model knows it's around 14 million in the city proper, 37 million in the greater metropolitan area) but might warrant a database lookup for an application requiring precise current figures. The right behavior depends on the application's accuracy requirements, which should be communicated through the system prompt.

Validating function calling models requires specialized metrics beyond standard perplexity or BLEU scores. These metrics assess both the syntactic correctness and semantic appropriateness of tool use:

• Schema adherence rate: Percentage of generated calls that parse as valid JSON and match the defined schema. This is a hard constraint; even minor syntax errors render the call unusable.
• Parameter accuracy: Correctness of extracted parameters (e.g., did the model correctly identify "Boston" as the location). This measures the model's ability to map natural language entities to structured fields.
• Tool selection accuracy: Frequency with which the model chooses the correct function from multiple options. This assesses the model's semantic understanding of tool purposes and boundaries.
• False positive rate: How often the model invokes tools unnecessarily for answerable questions. High false positive rates indicate poor calibration and lead to wasted computational resources.
• Hallucination rate: Frequency of invented parameter values or non-existent function calls. This measures the model's tendency to confabulate when faced with uncertainty.

These metrics are typically evaluated on held-out test sets with gold-standard annotations showing the correct tool sequence for each query. Evaluation should include adversarial examples designed to trick the model into incorrect tool selection or parameter hallucination. This tests resilience against edge cases.

The interaction between these metrics creates interesting trade-offs. Aggressive training on tool-required examples reduces the false positive rate (the model learns that direct questions should be answered directly), but may reduce schema adherence if the training data lacks sufficient format diversity. Evaluating on all metrics simultaneously and tracking their correlations helps identify which aspects of the training data need reinforcement.