Model Merging › DARE Merging

TIES improves over naive task arithmetic by handling sign conflicts, but it still includes all parameters that survive the trim step. DARE (Drop And Rescale) takes a different approach: randomly drop a large fraction of the task vector and rescale the remaining values to compensate. The name of the paper introducing DARE captures its spirit well: "Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch."

The method was introduced by Yu et al. in 2023. The paper's title is a reference to the video game mechanic where Mario absorbs power-ups without losing his prior abilities, which is an analogy for how model merging lets a base model absorb fine-tuned capabilities without forgetting its pretrained knowledge.

DARE is motivated by an empirical observation: fine-tuned models differ from their pretrained base in many small ways, but most of these differences are redundant. Many parameters can be set back to their pretrained values without noticeably harming the fine-tuned model's performance, as long as the remaining changed parameters are appropriately rescaled.

This observation is not obvious. You might expect that every nonzero entry in a task vector contributes something important, and that removing any of them would hurt performance. The empirical reality is different. Fine-tuning with gradient descent produces many small, correlated updates that collectively shift the model's behavior, but any individual small update contributes marginally. Setting 90% of the task vector entries back to zero while rescaling the remaining 10% preserves the model's task performance surprisingly well. The redundancy of fine-tuning updates is a property of the overparameterized regime in which modern language models operate.

Formally, DARE applies a random pruning mask $m_i \in \{0, 1\}^d$ to each task vector, where each entry is independently set to 0 with probability $p$ (the drop rate) and to 1 with probability $1 - p$:

$$\hat{\tau}_i = m_i \odot \tau_i$$

where:

• $\hat{\tau}_i$: the pruned task vector after applying the mask
• $m_i$: the binary random mask, each entry 1 with probability $1 - p$
• $\odot$: element-wise multiplication
• $p$: the drop rate, typically set to 0.9 or higher, meaning 90% or more of entries are dropped

The key difference from TIES's trim step is that DARE uses random sampling rather than magnitude-based selection. Instead of always keeping the top-magnitude entries, DARE draws a random mask and keeps whichever entries the mask selects. This randomness prevents the merge from systematically ignoring any particular parameter, which can be beneficial when the large-magnitude entries of different task vectors conflict with each other.

Dropping 90% of task vector entries would normally collapse the scale of the updates and effectively undo most of the fine-tuning. DARE compensates by rescaling the surviving values by $\frac{1}{1-p}$:

$$\tilde{\tau}_i = \frac{1}{1-p} \hat{\tau}_i = \frac{m_i}{1-p} \odot \tau_i$$

where:

• $\tilde{\tau}_i$: the rescaled, pruned task vector
• $\frac{1}{1-p}$: the rescaling factor that preserves expected magnitude, analogous to the inverse dropout scaling used in neural network training

This is mathematically identical to the dropout trick used during neural network training, where activations are randomly zeroed and then scaled up by the inverse survival probability to preserve expected values. In expectation, the rescaled pruned vector has the same mean as the original:

$$\mathbb{E}[\tilde{\tau}_i[j]] = \mathbb{E}\left[\frac{m_i[j]}{1-p}\right] \cdot \tau_i[j] = \frac{(1-p)}{1-p} \cdot \tau_i[j] = \tau_i[j]$$

So in expectation, DARE's pruned and rescaled task vector is unbiased relative to the original. Each individual realization will differ from the original because of the randomness, but on average, the vector preserves the full magnitude of the original task updates. This is what makes DARE safe to apply aggressively: you are not systematically reducing the scale of updates, only introducing sparsity while preserving the expected scale.

DARE is not itself a merging method but rather a preprocessing step applied to each task vector before merging. After applying DARE to each task vector independently, you can combine the sparsified vectors using any merging strategy, such as simple averaging or TIES.

The full DARE + TIES pipeline works as follows. First, compute task vectors $\tau_i = \theta_i - \theta_{\text{pretrained}}$ for each fine-tuned model. Second, apply DARE to each task vector: randomly drop $p$ fraction of entries and rescale the survivors by $\frac{1}{1-p}$. Third, apply TIES to the resulting sparse task vectors: trim any residual small values, elect signs by magnitude-weighted majority vote, and disjoint merge. Finally, add the merged task vector to the pretrained model: $\theta_{\text{merged}} = \theta_{\text{pretrained}} + \lambda \cdot \tau_{\text{merged}}$.

The combination often outperforms either DARE or TIES alone because the two methods address complementary sources of noise. DARE's random sparsification reduces redundancy by independently sparsifying each task vector, preventing the same small, correlated updates from accumulating across models. TIES's sign election then resolves any remaining conflicts between the surviving nonzero entries. The two steps together produce a cleaner merged task vector with less noise and more reliable signal.

The fact that dropping 90% of task vector entries still preserves most model behavior is initially surprising. Why should such aggressive pruning work?

The explanation connects to overparameterization. Modern language models have far more parameters than strictly necessary to represent any single task. When fine-tuning adjusts many parameters, only a fraction of them are truly essential to the task's performance. The majority of nonzero entries in the task vector represent small, redundant adjustments that collectively push the model in the right direction but individually contribute little. DARE acts as a noise filter, discarding many of the small, incidental changes that fine-tuning makes and retaining only the most important structural modifications.

This finding aligns with the observations from sparse fine-tuning literature, where methods like LoRA demonstrate that you can fine-tune a large model using only a small number of low-rank updates and achieve performance comparable to full fine-tuning. The effective rank of the fine-tuning update is much lower than the full parameter count, which means most of the task vector lives in a low-dimensional subspace. DARE exploits this by randomly sampling from the full parameter space, statistically covering the important subspace while discarding much of the noise.

The randomness in DARE also serves a practical purpose for merging multiple models. When two task vectors have conflicting values at the same parameter position, DARE randomly zeros out one or both of them with high probability. This stochastic conflict resolution is less systematic than TIES's sign election but is computationally cheaper and can reduce interference in ways that deterministic methods might miss.