PEFT Beyond LoRA (2024) › The Solution

Researchers in 2024 developed several advanced parameter-efficient fine-tuning techniques that addressed LoRA's limitations through more adaptive and sophisticated approaches. These methods shared a common philosophy of intelligently allocating adaptation capacity rather than distributing it uniformly, and they explored alternative parameterization schemes that could capture richer update patterns while maintaining efficiency.

AdaLoRA, developed by researchers at Microsoft and published in early 2024, addressed LoRA's fixed-rank limitation by adaptively allocating rank across different layers based on their importance. Instead of using the same rank for all layers, AdaLoRA dynamically adjusted the rank of each layer's adaptation during training. Layers that contributed more to task performance received higher ranks, while less important layers were allocated lower ranks or even zero rank, effectively pruning unnecessary adaptations.

The method achieved this through an importance-based ranking mechanism. AdaLoRA initialized adapters with higher ranks than needed, then used importance scores derived from gradient magnitudes or other metrics to prune away less important components. During training, the method continuously evaluated the contribution of each rank component and could reduce rank for layers that didn't need high capacity while increasing it for more critical layers. This adaptive allocation ensured that adaptation capacity was concentrated where it mattered most, improving efficiency without sacrificing performance.

AdaLoRA also introduced a novel decomposition that separated singular values from their corresponding vectors, enabling more fine-grained rank adjustment. Rather than treating the rank-$r$ adaptation as a fixed low-rank matrix product, AdaLoRA used a singular value decomposition (SVD) format that allowed individual singular values to be pruned. This decomposition provided additional flexibility in rank allocation, as the method could reduce effective rank by pruning small singular values while preserving the most important adaptation directions.

DoRA, proposed by researchers later in 2024, addressed LoRA's limitations by decomposing weight updates into magnitude and direction components. Instead of directly approximating weight updates through low-rank matrices, DoRA decomposed the updated weight matrix $W'$ into a magnitude vector $m$ and a direction matrix that was approximated using LoRA. The update took the form $W' = m \cdot \frac{W + \Delta W}{\|W + \Delta W\|_c}$, where $\Delta W$ was the LoRA approximation and $m$ was a trainable magnitude vector.

This decomposition recognized that weight updates involved both changing the direction of weight vectors and adjusting their magnitudes. LoRA primarily captured directional changes through its low-rank approximation, but magnitude adjustments could be important for some adaptations. By explicitly modeling magnitude separately, DoRA could learn more effective updates while maintaining parameter efficiency. The magnitude vector added relatively few parameters compared to the full weight matrix, but it provided additional flexibility that improved adaptation quality.

DoRA's approach proved particularly effective for tasks requiring significant domain shifts or specialized behaviors. The explicit magnitude control allowed the method to scale weight updates appropriately for different layers and components, while the directional updates captured through LoRA maintained efficiency. This combination enabled DoRA to achieve better performance than standard LoRA on several benchmarks. This showed that parameter-efficient fine-tuning could be improved through more advanced update parameterizations.

VeRA introduced a fundamentally different approach to parameter-efficient fine-tuning by using shared random matrices across all layers. Instead of learning separate low-rank matrices for each layer, VeRA used a single pair of frozen random matrices shared across the entire model, along with small learnable scaling vectors for each layer. This approach dramatically reduced the number of trainable parameters while still enabling layer-specific adaptations through the scaling vectors.

The method worked by defining adaptations as $\Delta W_l = d_l \circ (B \cdot A)$, where $B$ and $A$ were frozen random matrices shared across all layers, and $d_l$ was a learnable vector specific to layer $l$. The element-wise multiplication with $d_l$ allowed each layer to scale the shared random projection appropriately for its needs. This design reduced trainable parameters by orders of magnitude compared to standard LoRA, as only the small scaling vectors needed to be learned rather than full low-rank matrices for each layer.

VeRA's approach built on the insight that many fine-tuning tasks required relatively simple scaling adjustments to random projections rather than complex learned transformations. By sharing random matrices across layers, the method could capture common adaptation patterns efficiently while still allowing layer-specific customization through scaling. This technique demonstrated that parameter efficiency could be pushed even further by questioning fundamental assumptions about how adaptations should be structured.

Several other methods emerged throughout 2024 that explored different dimensions of parameter-efficient fine-tuning. RSLoRA (Rank-Stabilized LoRA) addressed training instability issues in LoRA by modifying the initialization and training dynamics. The method stabilized training by carefully initializing adapter matrices and adjusting learning rates to prevent rank collapse, where the effective rank of adaptations decreased during training, reducing adaptation capacity.

LoftQ (LoRA-Fine-Tuning-aware Quantization) combined quantization with LoRA in a way that was aware of the fine-tuning process. Rather than quantizing the base model independently and then applying LoRA, LoftQ jointly optimized quantization and adapter initialization to minimize quantization errors in the context of fine-tuning. This approach improved performance when combining quantization with LoRA, addressing scenarios where QLoRA's approach might not be optimal.

These methods collectively demonstrated that parameter-efficient fine-tuning was a rich area for innovation, with multiple dimensions along which improvements could be made. Rank allocation, update parameterization, parameter sharing, and training dynamics all offered opportunities for optimization that could improve efficiency or performance while maintaining the core benefits of parameter-efficient approaches.