Model Merging Applications › Merging Evaluation

Evaluation is the bottleneck in practical merging work. Unlike fine-tuning, where loss curves provide continuous feedback, merging gives you a single candidate model per configuration. Poor evaluation leads to poor configuration choices, wasting the effort of the merge itself. Getting evaluation right is not a secondary concern, it is the core engineering challenge.

The first requirement for reliable merging evaluation is broad benchmark coverage. A single benchmark number is not sufficient because merged models often show asymmetric performance changes: they may improve on tasks related to one source model's fine-tune while regressing on tasks related to another. This asymmetry is invisible if you only measure one task.

A practical evaluation suite for a merged model should include:

• At least one benchmark for each source model's intended capability, measuring whether the merge preserved each fine-tune's core contribution
• A general language understanding benchmark (such as MMLU or HellaSwag) to detect global degradation in base capabilities, which can happen when conflicting task vectors suppress general knowledge
• A generation quality metric (perplexity, BERTScore, or MT-bench) to catch cases where the model generates incoherent text despite acceptable task scores
• An instruction-following evaluation if the models were instruction-tuned, because instruction-following is sensitive to the fine-tuning mix and can regress while task-specific metrics remain stable

The composition of this suite will vary by domain, but the principle holds: you need to measure all capabilities you care about, not only the capability you are trying to add.

Raw benchmark numbers are hard to compare across different evaluation suites and scales. A score of 78% on one benchmark and 63% on another does not immediately tell you whether the merge succeeded or failed relative to what was achievable. A useful normalization is:

$$\text{normalized score}_t = \frac{\text{score}_{\text{merged}}(t) - \text{score}_{\text{base}}(t)}{\text{score}_{\text{ft}}(t) - \text{score}_{\text{base}}(t)}$$

where:

• $\text{score}_{\text{merged}}(t)$: the merged model's score on task $t$
• $\text{score}_{\text{base}}(t)$: the base model's score on task $t$ (the floor)
• $\text{score}_{\text{ft}}(t)$: the task-specific fine-tuned model's score on task $t$ (the ceiling)

This normalized score equals 1.0 if the merge perfectly preserves the fine-tune's performance, 0.0 if it fully reverts to the base, and can exceed 1.0 in rare cases where merging produces emergent improvement above the fine-tune's original level. Averaging normalized scores across all tasks gives an overall efficiency measure for the merge configuration.

The formula has an important interpretation: it measures what fraction of the fine-tuning benefit you retained. If the fine-tune improved task performance from 45% (base) to 75% (fine-tune), a normalized score of 0.8 means the merged model achieves $45 + 0.8 \times (75 - 45) = 69\%$. You retained 80% of the improvement the fine-tuning achieved, which is typically acceptable.

In practice, scores above 0.8 are considered good for multi-task merges, scores between 0.5 and 0.8 indicate moderate degradation that may be acceptable depending on deployment requirements, and scores below 0.5 suggest the merge is not working well and should be revisited with different algorithms or coefficients.

For a merge of two models, there is typically a tradeoff between performance on Task 1 and performance on Task 2. As $\lambda$ increases (weighting Task 1's fine-tune more heavily), Task 1 performance rises and Task 2 performance falls.

Plotting this tradeoff across all tested $\lambda$ values traces a curve that approximates the Pareto frontier of achievable configurations. The Pareto-optimal configurations are those where you cannot improve performance on one task without degrading the other. The goal of merging evaluation is to locate the Pareto frontier and then choose the configuration that best matches your deployment priorities.

In practice, the Pareto frontier for well-chosen model pairs lies above the diagonal (above the line connecting "all Task 1" to "all Task 2"), meaning merged models can be better at both tasks simultaneously than a simple interpolation would predict. This super-additive region is what makes merging useful rather than just a weighted average.

The diagonal baseline represents a model that selects outputs from either the Task 1 fine-tune or the Task 2 fine-tune at random, weighted by $\lambda$. Any merge configuration that achieves better performance on both tasks than this diagonal baseline produces a synergistic benefit from combining the two models' weights. The degree to which the Pareto frontier lies above this diagonal quantifies how much benefit the merge is providing beyond simple model selection.

When the Pareto frontier lies below the diagonal, the merge is actively harmful: you would do better by routing queries to one of the two source models based on task detection rather than using the merged model at all. This failure pattern typically occurs when the two tasks have high interference and the chosen merging algorithm is not handling it effectively.

Many benchmarks are slow to evaluate. Running MMLU, for example, requires thousands of forward passes with multi-choice completion. When searching over many merge configurations, you need faster proxies.

Two useful proxies have emerged from empirical experience with merging workflows.

Perplexity on task-specific data is fast and correlates reasonably well with downstream task performance. A merged model with lower perplexity on both tasks' held-out corpora is usually better than a merged model with higher perplexity on one, even before running the full benchmark. Computing perplexity requires a single forward pass per example with teacher-forced targets, which is faster than generating completions. On a single GPU, you can evaluate perplexity on 1,000 examples from each task in under a minute for a 7B-parameter model.

Calibration quality (the alignment between model confidence and actual accuracy) degrades in poorly merged models before task accuracy does. A merged model that becomes overconfident or underconfident is a signal of interference, even if task accuracy has not yet dropped measurably. Calibration can be measured with Expected Calibration Error (ECE) on a small labeled validation set. High ECE in a candidate configuration is a reliable early warning sign that the merge is causing internal representational conflicts.

A practical three-stage evaluation workflow combines these proxies efficiently. In the first stage, compute perplexity on held-out samples from each task's corpus for all candidate configurations. This takes seconds per configuration and immediately eliminates the worst options. In the second stage, run abbreviated benchmark evaluations (using 10-20% of benchmark examples) on the remaining top-ranked configurations. This narrows the candidate set further at moderate cost. In the third stage, run full benchmark evaluations only on the top three to five configurations from stage two. This staged approach reduces total evaluation compute by an order of magnitude compared to running full benchmarks on every configuration.

The staged approach also has the practical benefit of providing early stopping: if the first-stage perplexity numbers show that all configurations degrade significantly compared to the source models, you can abort the search and reconsider whether the chosen task pair is a good merging candidate, rather than running expensive benchmarks only to discover the same conclusion.

Beyond task-specific performance metrics, merged models can exhibit failure modes that do not show up in standard benchmarks but become apparent in production. Three categories of emergent failure are worth monitoring for.

Consistency failures occur when a merged model gives contradictory answers to the same question phrased differently. A model that correctly answers "Paris is the capital of France" as a multiple-choice question but hallucinates "Lyon" when asked in open-ended generation may have merged models with conflicting factual associations for that piece of knowledge. Consistency can be tested by asking the same factual queries in multiple formats and checking agreement.

Style inconsistency is a failure mode specific to style merges, where the model switches between two styles within a single long-form output. The early tokens in a completion may exhibit one style while later tokens shift to the other, because the model's internal state has evolved in a way that activates a different style register. This can be detected by splitting long outputs into chunks and running the style classifier on each chunk separately, checking for within-output variance.

Instruction conflict occurs when merged instruction-tuned models have conflicting expectations about how to interpret certain prompt patterns. One source model may have been trained to refuse certain request types with a specific decline phrasing, while another may have been trained to fulfill similar requests in a different context. The merged model may exhibit unpredictable behavior on borderline requests that fall in the overlap region of the two training distributions.