PEFT Comparison › Parameter Efficiency Comparison

PEFT methods adapt large models by updating only a small fraction of parameters. To make a useful comparison, we need concrete metrics rather than qualitative descriptions. Examining the parameter counts reveals how methods differ in memory and storage requirements, and these numbers connect directly to the hardware constraints you'll face in practice.

The central question is not just "how few parameters can a method use?" but "how efficiently do those parameters capture the necessary adaptation?" A method that uses 1% of parameters but reaches only 80% of full fine-tuning performance is less parameter-efficient in a practical sense than a method that uses 0.3% of parameters and reaches 97% performance. Throughout this section, keep in mind that parameter count is a proxy for what we care about: the combination of hardware requirements, storage costs, and training time that determines whether a method is practical for your situation.

We will work through the parameter calculations for each method applied to a common reference model, establishing a shared baseline that allows direct comparison. The numbers we derive here are not just theoretical exercises. They explain the real-world constraints you encounter when choosing between a 16GB GPU and a 48GB GPU, or between running five experiments in parallel versus two.

Let's establish concrete numbers for a typical 7B parameter model like LLaMA-2-7B with hidden dimension $d = 4096$, 32 layers, and 32 attention heads. Calculating the parameters for each method shows the tradeoffs and suitability for different hardware constraints.

LoRA approximates weight updates by decomposing them into two low-rank matrices, $A$ and $B$, such that $\Delta W = BA$. This design relies on the hypothesis that weight updates have a low "intrinsic rank," meaning the necessary adaptations exist in a low-dimensional subspace. The key insight motivating this approach is that while the original weight matrices are enormous, the changes needed to adapt a model to a new task may be far simpler in structure. If we can capture these changes using matrices with reduced dimensionality, we reach large parameter savings without sacrificing the model's ability to learn the task. For a layer with hidden dimension $d$ and rank $r$, the number of parameters is $d \times r + r \times d = 2dr$. When applied to all four attention projections ($W_Q, W_K, W_V, W_O$), we sum across all layers to get the total trainable parameter count.

The total number of trainable parameters for LoRA is calculated by summing the parameters of the low-rank matrices $A$ and $B$ across all target modules and layers. This calculation reveals how the rank parameter $r$ is a direct control knob for the tradeoff between model capacity and parameter count:

$$\text{LoRA params} = 4 \times (2 \times d \times r) \times L$$

where:

• $4$: the number of target modules per layer (Query, Key, Value, Output)
• $2 \times d \times r$: parameters for matrices $A$ and $B$ in one module (since each has size $d \times r$ or $r \times d$)
• $d$: hidden dimension ($4096$)
• $r$: rank of the low-rank adapters ($16$)
• $L$: number of layers ($32$)

For our LLaMA-2-7B setup, this yields:

$$\begin{aligned} \text{LoRA params} &= 4 \times 2 \times 4096 \times 16 \times 32 \\ &\approx 16.8\text{M} \end{aligned}$$

This result is striking: we can adapt a 7 billion parameter model by training only 16.8 million parameters, a reduction of over 400 times. The rank of 16 is deliberately chosen to be small enough to give substantial parameter savings while still giving sufficient capacity for most adaptation tasks. If the task proves to be complex, increasing the rank to 32 or 64 gives more capacity while still training only a small fraction of the model's parameters.

Adapters insert bottleneck modules, consisting of a down-projection and an up-projection, after both the attention and feed-forward (FFN) layers. This bottleneck architecture compresses inputs into a lower-dimensional representation before expanding them back, reducing parameter count while letting non-linear adaptation. The basic idea behind adapters is that task-specific information can be "filtered" through a narrow bottleneck, forcing the model to learn only the most needed transformations. This compression-expansion pattern also introduces non-linearity through activation functions, giving adapters additional representational power compared to linear methods like LoRA. With bottleneck dimension $d_{\text{bottleneck}} = 64$, we calculate the parameters for one adapter:

We calculate the parameter count by summing the weights of the down-projection and up-projection matrices for each adapter. The symmetry of this calculation, with equal contributions from each projection direction, shows the balanced nature of the bottleneck design:

$$\text{Adapter params} = 2 \times (d \times d_{\text{bottleneck}} + d_{\text{bottleneck}} \times d) \times L$$

where:

• $2$: the number of adapters per layer (one inserted after Attention, one after FFN)
• $d$: the hidden dimension of the model ($4096$)
• $d_{\text{bottleneck}}$: the size of the bottleneck dimension ($64$)
• $d \times d_{\text{bottleneck}}$: the number of parameters in the down-projection matrix
• $d_{\text{bottleneck}} \times d$: the number of parameters in the up-projection matrix
• $L$: the number of layers in the model ($32$)

Plugging in the values:

$$\begin{aligned} \text{Adapter params} &= 2 \times (4096 \times 64 + 64 \times 4096) \times 32 \\ &\approx 33.6\text{M} \end{aligned}$$

Adapters require roughly twice as many parameters as LoRA with rank 16. This increased parameter count buys additional expressiveness through the non-linear activation function between the down and up projections, but comes at the cost of both storage and, as we'll see later, inference overhead. The bottleneck dimension of 64 was chosen here to be illustrative; in practice, practitioners often tune this value just as they would tune LoRA's rank.

Prefix Tuning prepends learnable vectors to the keys and values at every layer. These virtual tokens effectively "steer" the attention mechanism at every depth, guiding the model's internal processing without modifying any weights. Rather than modifying the model's weights directly, prefix tuning modifies the context that the model attends to at every layer. Optimizing key-value pairs at every layer influences attention flow without changing the transformation matrices. With prefix length $l = 20$, we can calculate the total parameter count.

The trainable parameters consist of the virtual token embeddings added to the Key and Value matrices at each layer. This formulation makes clear why prefix tuning affects only the attention mechanism: we're adding content for the model to attend to, not modifying how it processes that content:

$$\text{Prefix params} = l \times d \times 2 \times L$$

where:

• $l$: the prefix length (number of virtual tokens, set to $20$)
• $d$: the hidden dimension ($4096$)
• $2$: a factor accounting for application to both Key and Value matrices
• $L$: the number of layers ($32$)

Substituting the values:

$$\begin{aligned} \text{Prefix params} &= 20 \times 4096 \times 2 \times 32 \\ &\approx 5.2\text{M} \end{aligned}$$

The parameter count for Prefix Tuning is lower than LoRA despite operating at every layer. This efficiency comes from the method's focused scope: it only influences the attention mechanism through the key-value channel, leaving all other computations unchanged. However, this scope also explains one of Prefix Tuning's key weaknesses, which we'll analyze when we examine task performance.

Prompt Tuning adds learnable embeddings only at the input layer. Unlike discrete text prompts, these continuous embeddings are optimized directly via backpropagation to find the most effective task trigger. This approach is perhaps the simplest possible form of model adaptation: we're in effect learning a better way to "ask" the model to perform our task. The continuous nature of these embeddings allows optimization to find representations that may have no natural language equivalent, potentially discovering more effective task specifications than any human-written prompt could give. With prompt length $p = 100$:

The parameters come solely from the learnable embeddings prepended to the input layer. This single-layer application is what gives Prompt Tuning its remarkable parameter efficiency, though it also limits the method's ability to influence deep model computations:

$$\text{Prompt params} = p \times d$$

where:

• $p$: the prompt length (number of virtual tokens, set to $100$)
• $d$: the embedding dimension ($4096$)

Calculating the total:

$$\begin{aligned} \text{Prompt params} &= 100 \times 4096 \\ &\approx 0.4\text{M} \end{aligned}$$

With fewer than half a million parameters, Prompt Tuning reaches the smallest footprint of any method we've examined. This extreme efficiency makes it ideal for scenarios where storage or memory is at an absolute premium, though as we'll see in the performance analysis, this efficiency comes with real tradeoffs.

IA³ introduces learned vectors that scale the activations element-wise. This acts like a feature equalizer, selectively amplifying or suppressing specific activation channels based on what the task requires. IA³ assumes the base model computes useful features and only needs to adjust their relative importance for different tasks. Rather than learning new transformations, IA³ learns which existing computations to emphasize and which to diminish. IA³ is effective because it uses pre-trained representations: the model has already learned to compute real features, and IA³ simply re-weights them for the task at hand. We calculate the parameter count by summing the scaling vectors for the Key, Value, and FFN intermediate activations:

$$\text{IA}^3 \text{ params} = (d_k + d_v + d_{\text{ff}}) \times L$$

where:

• $d_k, d_v$: the dimension of scaling vectors for attention keys and values ($4096$, matching the hidden dimension)
• $d_{\text{ff}}$: the dimension of the scaling vector for the FFN expansion layer ($11008$)
• $L$: the number of layers ($32$)

This results in a very small number of parameters:

$$\begin{aligned} \text{IA}^3 \text{ params} &= (4096 + 4096 + 11008) \times 32 \\ &\approx 0.61\text{M} \end{aligned}$$

The parameter count for IA³ is remarkably low, just over half a million for a 7B model. This efficiency stems from the element-wise nature of the scaling operation: we need only one scalar per activation dimension, rather than full transformation matrices. The tradeoff is that IA³ can only rescale existing activations, not compute fundamentally new features.

Parameter count alone doesn't tell the full story. During training, memory consumption includes model weights, optimizer states, gradients, and activations. Understanding the complete memory picture is needed for determining which methods will fit on your hardware. A method might train fewer parameters but still require substantial memory for other components of the training process.

Think of memory during training as having four distinct buckets that must all fit simultaneously into GPU VRAM. The first bucket holds the model weights themselves, which you must keep loaded at all times regardless of how many parameters you're updating. The second bucket holds the optimizer states, which for AdamW include two floating-point values per trainable parameter (the first and second moment estimates). The third bucket holds the gradients computed during backpropagation, which require storage for every trainable parameter. The fourth bucket holds the activations saved during the forward pass that are needed for gradient computation during the backward pass. PEFT methods reduce the second and third buckets to near zero for frozen parameters, but the first bucket remains unchanged.

For full fine-tuning with AdamW, memory is consumed by weights, optimizer states, gradients, and activations. Assuming mixed precision training, we can estimate the total memory requirement by analyzing the contribution of each component:

$$\text{Memory}_{\text{full}} = \text{weights} + \text{optimizer} + \text{gradients} + \text{activations}$$

where:

• weights: memory for the model parameters (usually FP16), consuming 2 bytes per parameter
• optimizer: memory for optimizer states (momentum and variance in AdamW), consuming approximately 8 bytes per parameter in FP32
• gradients: memory for storing gradients during backpropagation, consuming approximately 4 bytes per parameter
• activations: memory for intermediate outputs needed for the backward pass, which scales with batch size and sequence length

Using a simplified estimation model:

$$\begin{aligned} \text{Memory}_{\text{full}} &\approx 2P + 4P + 4P + 2P \\ &= 12P \text{ bytes} \end{aligned}$$

where:

• $P$: the total number of model parameters
• $2P$: FP16 model weights ($2$ bytes per parameter)
• $4P$: optimizer states (simplified estimate for AdamW first and second moments)
• $4P$: gradients (FP32 or mixed precision)
• $2P$: activations (approximate average per layer)

For a 7B model, this works out to approximately $12 \times 7 \times 10^9 \approx 84$ GB of VRAM. Full fine-tuning is memory-intensive because requirements scale linearly with parameters, and each parameter needs storage for multiple values. No consumer GPU can come close to fitting these requirements.

PEFT methods drastically reduce memory by freezing the base model and training only the adapter parameters. The key insight is that frozen parameters need only their weights stored, not optimizer states or gradients. This is the most important memory saving PEFT gives: by freezing 99%+ of parameters, we eliminate the need to store optimizer states and gradients for those parameters, reducing the total memory requirement by a factor of 5-6x. With LoRA (rank 16):

For LoRA, we only need to store the optimizer states and gradients for the small set of trainable parameters, while the base model remains frozen in low precision. This separation between frozen and trainable parameters is the basic mechanism through which all PEFT methods reach their memory efficiency:

$$\text{Memory}_{\text{LoRA}} \approx 2P_{\text{frozen}} + 12P_{\text{trainable}}$$

where:

• $P_{\text{frozen}}$: the number of frozen base model parameters ($7$ billion)
• $P_{\text{trainable}}$: the number of trainable LoRA parameters ($16.8$ million)
• $2$: bytes per frozen parameter (FP16 weights only, no optimizer state or gradient needed)
• $12$: the memory factor for trainable parameters (weights + optimizer + gradients + activations)

Applying this to our example:

$$\begin{aligned} \text{Memory}_{\text{LoRA}} &\approx (2 \times 7 \times 10^9) + (12 \times 16.8 \times 10^6) \\ &\approx 14 \text{ GB} + 0.2 \text{ GB} \\ &= 14.2 \text{ GB} \end{aligned}$$

This is why a 7B model that requires 80+ GB for full fine-tuning can be fine-tuned with LoRA on a single 24GB GPU. Memory savings make fine-tuning accessible; LoRA lets adaptation on hardware unsuitable for full fine-tuning. This shift in memory requirements lets you to adapt large models for your specific needs without expensive compute clusters or cloud GPU instances. The practical implication of this reduction cannot be overstated: it transformed large model fine-tuning from an activity requiring specialized infrastructure into something a graduate student with a gaming GPU can do.