PEFT Motivation › PEFT Motivation

In Part XXXIV, we explored full fine-tuning: updating all parameters of a pre-trained model to adapt it for a specific downstream task. The mechanics are conceptually simple. You take a pre-trained model, expose it to task-specific training data, and use gradient descent to nudge every single weight toward better task performance. Full fine-tuning produces excellent results, and for many years it was the unquestioned standard for specializing language models. But as models have grown from millions to billions of parameters, a severe practical problem has emerged, and that problem is now impossible to ignore. Training a separate 70-billion-parameter model for each task has become prohibitively expensive. Storing and deploying each copy adds further cost, even for organizations with substantial resources.

Think of full fine-tuning at scale as buying an entirely new house every time you want to redecorate a room. You get exactly the aesthetic you want, but the cost is wildly out of proportion to the outcome. Parameter-Efficient Fine-Tuning (PEFT) addresses this by asking a more targeted question: do we really need to change every wall, every floor, every foundation just to repaint one bedroom? PEFT methods update only a small fraction of parameters, typically less than 1% of the total, while keeping most of the model frozen. The result is a model that performs almost as well on the target task while requiring less compute and memory while using a fraction of the storage.

The economic case for PEFT follows directly from the cost of full fine-tuning at scale. A model with 7 billion parameters trained in 16-bit floating point occupies 13 GB of disk space. A model with 70 billion parameters requires 130 GB. When you multiply these numbers by the dozen or more specialized tasks a production AI system might need to handle, and then again by the training memory overhead, the costs become staggering. The hardware required to train a 70B model from scratch exceeds what most universities and startups or mid-sized companies can access, let alone run continuously across multiple task variants.

This chapter examines why PEFT has become essential for modern LLM deployment. We quantify the storage and compute costs of full fine-tuning at scale, explore the challenges of serving multiple specialized models in production, and understand the trade-offs between parameter efficiency and task performance. Importantly, we will build the conceptual foundation that makes the specific PEFT methods in subsequent chapters feel like natural solutions rather than arbitrary design choices. By the time we reach LoRA, prefix tuning, and adapter layers, you will already understand exactly what problem each technique is solving.