IA3 › Summary

IA3 offers a remarkably simple and efficient approach to parameter-efficient fine-tuning: instead of learning weight updates like LoRA, it learns to rescale existing activations. By learning three small vectors ($\ell_K$, $\ell_V$, and $\ell_{ff}$) that multiply keys, values, and feed-forward intermediate activations respectively, IA3 adapts pretrained models with sharply fewer parameters than any other common PEFT method. For a 7-billion-parameter model, IA3 introduces just 600,000 trainable parameters, roughly 0.01% of the total.

The main ideas to take away from this chapter:

• Rescaling as adaptation: Learned element-wise multiplication effectively adapts pretrained models by amplifying needed activation dimensions and suppressing irrelevant ones, without modifying any weight matrix.
• Multiplicative vs additive: IA3's multiplicative nature (multiplying activations) differs fundamentally from LoRA's additive nature (adding weight updates). This difference explains both IA3's stronger inductive bias for few-shot settings and its limited expressivity for complex adaptations.
• Extreme efficiency: IA3 requires $2d + d_{ff}$ parameters per layer (approximately $6d$ for typical architectures), compared to LoRA's $14rd$ parameters per layer. At rank 8, IA3 uses approximately 20 times fewer trainable parameters.
• Initialization as identity: Starting all rescaling vectors at $\mathbf{1}$ ensures the model begins training with exactly the pretrained behavior. This gives a stable and interpretable starting point.
• Limited expressivity: IA3's diagonal adaptation cannot learn rotations or projections; it can only adjust the magnitude of existing computations. This is the price of its efficiency and the source of its inductive bias.
• Practical trade-offs: IA3 shines in few-shot settings and memory-constrained deployments. LoRA is often preferable when more training data is available or when the task differs substantially from pretraining.

In the next chapter on Prefix Tuning, we explore yet another perspective on minimal parameter adaptation: rather than rescaling activations within the model, prefix tuning prepends learned "soft" tokens to the input sequence that steer the model's behavior through the standard attention mechanism. This contrasts with IA3's direct activation scaling and LoRA's weight matrix augmentation, completing our tour of the major paradigms for making large language models task-specific without retraining them from scratch.