Model Merging › Limitations and Practical Considerations

Model merging is a powerful idea, but it comes with important limitations that constrain where and how it can be applied. Understanding these limitations is essential for using the methods effectively and for recognizing when to reach for a different approach.

The most fundamental requirement is architectural compatibility. All models being merged must share exactly the same architecture: the same number of layers, the same layer dimensions, and the same parameter layout. You cannot merge a 7B model with a 13B model, or a model using rotary position embeddings with one using absolute position embeddings. In practice, this restricts merging to models derived from the same pretrained base, which is the dominant use case on platforms like Hugging Face where many practitioners fine-tune and share variants of Llama, Mistral, or Phi. The Hugging Face Hub hosts thousands of fine-tuned variants of popular base models, and model merging has become a standard technique for combining these variants into capable generalists.

Even when architectures match, semantic alignment of parameters is not guaranteed. Weight averaging and task arithmetic implicitly assume that the $j$-th parameter in model $A$ encodes the same abstract concept as the $j$-th parameter in model $B$. For models derived from the same pretrained checkpoint, this assumption generally holds because fine-tuning makes relatively small perturbations and the parameter indices retain their semantic roles from pretraining. For models trained from different initializations, even with the same architecture, the same index may encode completely unrelated things and averaging them is meaningless. This is why techniques like SLERP (spherical linear interpolation, which accounts for the curved geometry of weight space) and model permutation alignment (which finds permutations of hidden units that improve alignment before merging) are active research areas, addressing the harder case of merging models that were not trained from the same starting point.

Hyperparameter sensitivity is another practical challenge. The scaling coefficient $\lambda$ in task arithmetic and TIES, the trim percentage in TIES, and the drop rate in DARE all affect merge quality substantially. Finding good values requires either a validation set covering all tasks or grid search over the hyperparameter space. Automated merge optimization tools, including tools that treat the merge coefficients as learnable parameters and optimize them with a small amount of gradient descent on a validation set, have emerged to address this problem. But the field lacks reliable, transferable heuristics that work across models and tasks without tuning, which means practitioners typically need to experiment.

Model merging also does not solve the problem of fundamentally conflicting tasks. If two tasks require opposing model behaviors, their task vectors will be in direct conflict for many parameters. TIES's sign election will pick one direction and discard the other, effectively forcing the merged model to choose a winner rather than combining the capabilities. True multi-task capability in such cases requires either specialized training, a mixture-of-experts architecture where different experts handle different tasks, or task-routing mechanisms that select different parameter sets for different inputs. Model merging works best when the tasks being merged are complementary or at least orthogonal in their parameter usage, not when they pull in directly opposite directions.

There is also the question of measuring success. A merged model might perform better on average across tasks but worse on any individual task compared to the specialized fine-tune. Whether this is acceptable depends on the use case. If you need a model that handles ten different tasks reasonably well, an average improvement might be exactly what you want. If you need peak performance on one specific task, the specialized fine-tune is likely better. The benchmarks used to evaluate merged models matter, and cherry-picked benchmarks can make any method look better than it is.

Despite these limitations, model merging has had real practical impact. The Hugging Face Open LLM Leaderboard has consistently featured merged models near the top, with practitioners combining specialized fine-tunes in creative ways to outperform any individual component. Tools like mergekit have democratized the process, making it possible for practitioners without training infrastructure to produce competitive models simply by merging publicly available fine-tunes. mergekit supports all of the methods covered in this chapter, as well as more recent techniques like SLERP and linear interpolation with per-layer merge coefficients, and it handles the practical details of working with large model files without loading all parameters into memory simultaneously.

The emerging field of "model merging as capability composition" continues to push these methods forward, with newer techniques addressing gradient-based optimization of merge coefficients, layer-wise merging strategies where different layers use different merge settings, and combinations with quantization that produce merged models small enough to deploy on consumer hardware. The key observation that drives this entire field is that fine-tuned variants of the same base model form a rich combinatorial space of capabilities, and merging is the zero-cost way to explore that space.

When you want to apply model merging in practice, a few principles guide the choice of method and hyperparameters.

Start with the simplest method that might work. Weight averaging is the right baseline when you are merging checkpoints from different stages of a single training run or multiple fine-tunes with very similar objectives. Task arithmetic is the right starting point for merging models fine-tuned on distinct tasks, because the task vector formulation makes the operation interpretable and the scaling coefficient provides a straightforward knob.

Move to TIES when you are merging three or more models and notice that task arithmetic produces inconsistent results or that some tasks perform much worse in the merge than they do individually. TIES's sign election is most valuable when the models being merged have overlapping parameter sensitivities with opposing directions, which is common when fine-tuning on tasks that require different output styles or factual domains.

Consider DARE as a preprocessing step when you have reason to believe that your task vectors contain substantial noise, for instance when fine-tuning was performed with a high learning rate or for many epochs, or when the fine-tuned models show signs of overfitting. DARE's aggressive sparsification tends to reduce the footprint of each task vector, making the subsequent merge cleaner.

For the scaling coefficient $\lambda$, a sensible default is 0.4 when merging two models and 0.3 when merging three or more. These values are consistent with what practitioners report on the Hugging Face community forums and with the original paper recommendations. Always validate on a held-out set before deploying a merged model.