Model Merging › Weight Averaging

The simplest form of model merging is weight averaging, sometimes called model soup. Given two or more models that share the same architecture, you compute the element-wise average of their weight tensors. No gradients, no data, no hyperparameter search beyond which models to include. The operation is so simple it seems like it should not work, and yet it often does.

Formally, given $n$ models with weight vectors $\theta_1, \theta_2, \ldots, \theta_n$, the merged model has weights:

$$\theta_{\text{merged}} = \frac{1}{n} \sum_{i=1}^{n} \theta_i$$

where:

• $\theta_i$: the full vector of parameters (flattened weights of all layers) for model $i$
• $n$: the number of models being merged
• $\theta_{\text{merged}}$: the resulting merged weight vector

This is a surprisingly effective baseline. The intuition is that if each model has landed in a different region of the loss landscape after training from the same or similar initialization, averaging their weights can land in a region that inherits good properties from all of them. You are not picking a single best model; you are constructing a new model that sits in the middle of the space spanned by all the candidates.

The key insight is that modern large language models are typically fine-tuned from the same pretrained checkpoint. This shared starting point means all fine-tuned models begin in the same region of weight space. Each fine-tuning run makes relatively small, task-specific adjustments to the pretrained weights. The fine-tuned model is not a radically different object from the pretrained model; it is the same model with modest parameter perturbations applied on top. The resulting models are therefore close to each other in weight space, and their average is also likely to be near all of them.

This property, sometimes called linear mode connectivity when demonstrated rigorously, means that the loss surface along the linear path connecting two fine-tuned models is relatively flat. Averaging does not catastrophically fall off a cliff. The merged model sits in a region that all component models agree is reasonable, and it can generalize across their respective tasks because the fine-tuned behavior of each model is still approximately encoded in the merged weights.

Understanding why this connectivity holds requires thinking about what fine-tuning does to a large pretrained model. A 7-billion parameter model trained on hundreds of billions of tokens has developed internal representations that are dense, distributed, and redundant. No single parameter encodes a single concept. When fine-tuning on, say, code generation, the optimizer adjusts a distributed pattern across many parameters to shift the model's output distribution toward code. These shifts are modest relative to the scale of the pretrained weights. The average of two such modest shifts still preserves the pretrained model's core representations and typically satisfies both fine-tuning objectives reasonably well.

When models are trained from entirely different initializations, however, the picture changes. The same parameter index in model $A$ and model $B$ may encode completely different things, so averaging them destroys information without creating anything useful. Weight averaging requires that models share the same parameterization and, ideally, the same initialization. This is a hard requirement, not a soft one.

The model soup approach, introduced by Wortsman et al. in their 2022 paper "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time," takes simple averaging further by being selective about which models to include. Rather than averaging everything uniformly, you add models to the soup one at a time, keeping a model only if it improves the merged model's validation performance.

The procedure is:

1. Sort candidate models by their individual validation accuracy.
2. Start with the single best model as the initial soup.
3. Attempt to add the next-best model by averaging it into the current soup.
4. If the resulting soup performs better on validation, keep it; otherwise discard the candidate.
5. Continue until all candidates have been evaluated.

This greedy selection tends to produce better results than uniform averaging because it filters out models that move the soup in harmful directions. In practice, the candidate models might be checkpoints from different training runs, different hyperparameter configurations, or different amounts of fine-tuning on the same base model. Wortsman et al. demonstrated that model soups could achieve better accuracy and robustness than the best individual model, even outperforming ensembles of the same models in some settings, a remarkable result because inference cost is identical to running a single model.

The mechanism behind improved robustness is particularly interesting. Individual fine-tuned models often overfit to the specific details of their fine-tuning setup: the augmentation strategy, the learning rate schedule, the random seed. Averaging multiple such models smooths out these idiosyncrasies, producing a model that generalizes better to distribution shifts. The soup is less adapted to any single training artifact and more aligned with the shared signal across all models.

A natural extension of uniform averaging is weighted averaging, where different models contribute unequally to the final result:

$$\theta_{\text{merged}} = \sum_{i=1}^{n} w_i \theta_i$$

where:

• $w_i$: the weight assigned to model $i$, satisfying $\sum_{i=1}^{n} w_i = 1$ and $w_i \geq 0$
• $w_i$ can reflect validation performance, recency (later checkpoints weighted more), or domain relevance

Uniform averaging assigns $w_i = 1/n$ to all models. Weighted averaging lets you express preferences. If one model substantially outperforms others on the target domain, you might assign it a higher weight. If you trust one training run's hyperparameters more than another's, that run can contribute more to the final soup. Finding optimal weights is a convex optimization problem given a validation set, but the space of weights is large, and practitioners often default to simple heuristics like proportional assignment based on validation accuracy.