PEFT Comparison › Combining PEFT Methods

Michael BrenndoerferDecember 14, 20252 min read
Language AI HandbookAll chapters
Part 35 of 67Parameter-Efficient Fine-tuning
Chapter 265PEFT Comparison › Combining PEFT Methods
Reading section 8 of 13 · Combining PEFT Methods · 2 minSection 8 of 13 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 265 of 513

Section 8 of 13 · 2 min read

PEFT Comparison › Combining PEFT MethodsOne section · Sections in this chapter8 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

An emerging research direction explores combining multiple PEFT methods for improved performance and flexibility. The intuition is that different methods modify different aspects of model behavior, and their complementary strengths might compound when combined thoughtfully.

Think of combining PEFT methods like layering different types of adjustments to a complex system. A musician recording a song might adjust the EQ (frequency balance), apply compression (dynamic range), and add reverb (spatial characteristics) simultaneously. Each adjustment targets a different dimension of the audio, and combining them reaches results that no single adjustment could produce alone. Similarly, combining LoRA's weight modifications with activation-level adjustments from IA³ targets different computational pathways simultaneously.

LoRA combined with Adapters is one natural combination: apply LoRA to attention layers and adapters after FFN layers. This captures both low-rank attention modifications and complex post-FFN transformations that might require the non-linearity of the bottleneck architecture. The tradeoff is that you inherit the inference overhead of adapters alongside the benefits of LoRA.

Prompt Tuning combined with LoRA offers another interesting combination: use soft prompts for task-level conditioning while LoRA handles fine-grained adaptation. This separation of concerns, where the prompt handles high-level task specification and LoRA handles low-level behavioral adjustment, is particularly useful for instruction-following scenarios with multiple task types.

In[33]:
Code
# Example: Combining LoRA with different target modules
from peft import LoraConfig

# Comprehensive LoRA setup targeting both attention and FFN
comprehensive_lora = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",  # Attention projections
        "gate_proj",
        "up_proj",
        "down_proj",  # FFN projections (for LLaMA-style models)
    ],
    lora_dropout=0.05,
    bias="none",
)
Out[34]:
Console
Target modules: {'k_proj', 'gate_proj', 'q_proj', 'o_proj', 'down_proj', 'up_proj', 'v_proj'}
Total rank budget: 16 per module

Applying LoRA to both attention and FFN layers is the most common "combination" approach used in practice. It materially increases the parameter count compared to attention-only LoRA but gives much more complete adaptation capacity. For a 7B model with rank 16 applied to all seven module types listed above, the parameter count roughly doubles compared to attention-only LoRA, but remains well within the range that fits on consumer hardware with QLoRA.