Full Fine-tuning › Limitations and Practical Considerations

Full fine-tuning is powerful but comes with significant limitations that practitioners must understand. These limitations are basic constraints rather than minor inconveniences, and they have driven entire research directions aimed at making fine-tuning more efficient and accessible.

Computational cost scales directly with model size. Fine-tuning BERT-base requires moderate resources: a single GPU can complete training in minutes to hours depending on dataset size. Fine-tuning a 7B parameter model demands multiple high-end GPUs and can take days. Each forward and backward pass touches every parameter, and the cost grows linearly with the number of parameters. For the largest modern models, such as GPT-4 class systems with hundreds of billions of parameters, full fine-tuning is computationally impractical for most organizations. Even a single fine-tuning run would require thousands of GPU hours and significant infrastructure.

The memory requirements compound the computational challenge. For each parameter of size $p$ bytes, Adam stores the parameter itself ($p$ bytes), a first-moment buffer ($p$ bytes), and a second-moment buffer ($p$ bytes), totaling $3p$ bytes just for parameters and optimizer states. For a 7B parameter model in 16-bit precision, this amounts to approximately 42 GB, which exceeds the memory of any consumer GPU and most professional GPUs. Practitioners working with large models must use distributed training across multiple GPUs, adding complexity and infrastructure cost. This constraint is precisely what motivates the parameter-efficient fine-tuning approaches we will explore in Part XXXV of this handbook.

Storage requirements compound when serving multiple tasks. Each fine-tuned model is a complete copy of the original. If you fine-tune BERT for 10 different tasks, you need to store 10 complete models, each at full size. For a 3B parameter model at 16-bit precision, each task requires roughly 6 GB of storage (3 billion parameters times 2 bytes per parameter). Ten tasks require 60 GB, a hundred tasks require 600 GB, and so on. This creates significant infrastructure challenges for multi-task deployments, especially in production environments where models must be loaded and served with low latency. The storage cost alone can make full fine-tuning impractical for applications requiring many specialized variants of a model.

Catastrophic forgetting remains a basic challenge. As the model adapts to new data, it can lose capabilities from pre-training. A model fine-tuned heavily on medical text might perform worse on general language understanding than the original. The fine-tuning updates that improve performance on the target task necessarily move parameters away from the configurations that encoded general capabilities. This is not a bug but a consequence of gradient descent: the optimizer minimizes the target task loss, not the pre-training loss, so parameters shift toward configurations that favor the new task at the expense of old capabilities. We will examine this phenomenon in detail in the next chapter, covering why it occurs and techniques to mitigate it.

The magnitude of catastrophic forgetting depends on the learning rate, the number of epochs, and the similarity between the pre-training distribution and the fine-tuning distribution. Large learning rates and many epochs increase forgetting. Fine-tuning on a distribution similar to pre-training causes less forgetting than fine-tuning on a highly specialized domain. Understanding these dependencies helps you calibrate your fine-tuning configuration to reach good task performance while minimizing capability degradation.

Despite these limitations, full fine-tuning remains the standard for maximum task performance. When you have sufficient compute and storage, along with enough task-specific data, full fine-tuning typically outperforms parameter-efficient alternatives. The entire model can adapt its representations to the target domain, using every parameter toward the task objective. Parameter-efficient methods give useful approximations, but they constrain the space of representational changes the model can make, which sometimes limits their ceiling performance. When maximum accuracy is the goal and resources permit, full fine-tuning is the right choice.