PEFT Comparison › Computational Overhead Analysis

Beyond memory, PEFT methods differ in their computational costs during training and inference. These differences can materially impact both development iteration speed and production serving costs. A complete understanding of PEFT tradeoffs requires examining parameter counts, performance, and the time and compute required to train and deploy adapted models.

The computational picture has two distinct phases: training time and inference time. These phases have very different cost structures. During training, you typically run it once (or a few times with different hyperparameters), so the absolute training time matters less than during inference, which runs continuously in production. A method that takes 50% longer to train but adds zero inference overhead may be far more economical overall than a method that trains quickly but adds 15% latency to every prediction.

Training throughput depends on several factors that interact in complex ways. Understanding these interactions helps explain why methods with fewer trainable parameters don't always train proportionally faster.

The factors that determine training speed are:

1. Forward pass overhead: Added computation from adapter layers or modified attention. Methods that introduce new layers (Adapters) or extend sequence length (Prefix Tuning) increase forward pass time, even when those additions are small in absolute terms.
2. Backward pass scope: How many parameters receive gradients. Methods with fewer trainable parameters compute fewer gradients and update fewer optimizer states, reducing backward pass time materially.
3. Memory access patterns: Sequential access (Adapters) vs. broadcasted scaling (IA³). Memory-efficient operations can give substantial speedups even when parameter counts are similar. IA³'s element-wise multiplication is particularly cache-friendly compared to the matrix multiplications required by LoRA adapters.

Throughput improves primarily from reduced gradient computation. Prompt Tuning is fastest because gradients flow through only 100 embedding vectors, making the backward pass nearly instantaneous relative to full fine-tuning. The backward pass is faster because most parameters are frozen and require no gradient computation. These speed improvements, while less large than the memory savings, materially accelerate the iterative development cycle when you need to run many experiments to identify the right hyperparameters.

A necessary distinction separates PEFT methods at inference time: some add persistent overhead while others are "free" after training. This distinction has major implications for production deployments, where inference costs often dominate the total cost of ownership for ML systems. A model that serves millions of requests per day at 1.15x the base latency costs 15% more in compute than the base model, which at scale translates to substantial ongoing infrastructure expenses.

The key property that determines inference impact is whether the method's adaptations can be merged into the base model weights, eliminating all traces of the adaptation at serving time:

Zero inference overhead (mergeable):

• LoRA: Weights can be merged ($W + BA$) after training. The low-rank decomposition is purely a training-time construct; once the adapter is trained, we can precompute the full weight update and fold it into the base model weights. After merging, the serving model is identical in structure to the original.
• IA³: Scaling vectors can be absorbed into the preceding linear transformation's weights through a simple element-wise multiplication of the weight matrix columns. The result is a slightly modified weight matrix with identical inference behavior.
• Prompt Tuning: Only adds constant prefix tokens, which have minimal impact on inference time (typically less than 5%) since they are constant vectors that add a small fixed overhead per sequence.

Persistent inference overhead:

• Adapters: Must execute additional layers at every forward pass. The bottleneck computations cannot be removed because they involve non-linear activations that cannot be merged with surrounding linear operations. Every request pays the computational cost of the adapter's down-projection, non-linearity, and up-projection.
• Prefix Tuning: Increases the effective sequence length by the prefix size at every layer. The prefix tokens add to the key-value cache and increase attention computation proportional to the prefix length, which can be significant for long-context applications.