Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

PEFT Motivation › PEFT Efficiency

Michael BrenndoerferNovember 29, 202517 min read

Language AI HandbookAll chapters

Part 35 of 67Parameter-Efficient Fine-tuning

Chapter 254PEFT Motivation › PEFT Efficiency

Reading section 4 of 10 · PEFT Efficiency · 17 minSection 4 of 10 · 17 min

Section 4 of 10 · 17 min read

PEFT Motivation › PEFT EfficiencyOne section · Sections in this chapter4 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The challenges outlined above, including the storage multiplication problem, the enormous training memory requirements, and the operational complexity of serving multiple large models, point toward a fundamental question: is it really necessary to modify every single parameter to adapt a model to a new task? The answer, it turns out, is no, and the reasoning behind that answer is conceptually important.

When a model is pre-trained on hundreds of billions of tokens, it develops a rich internal representation of language: syntax, semantics, world knowledge, and reasoning patterns. These representations are encoded in the billions of weights. When you fine-tune for a specific task, you are not teaching the model a fundamentally new way to process language. You are teaching it to apply its existing capabilities in a particular way, to a particular domain, with a particular output format. This is a much smaller update than full pre-training suggests, and it should require updating far fewer parameters.

Think of it this way: a doctor who has spent years learning medicine does not need to relearn anatomy when they specialize in cardiology. They need to focus their existing knowledge, develop new associations in a narrow domain, and practice a specific set of skills. The vast majority of their medical training remains intact and relevant. PEFT is the training analogy for this kind of specialization: preserve the broadly learned knowledge, update only the parts that need to change for the new domain.

Parameter-efficient fine-tuning offers a fundamentally different approach: instead of modifying all model weights, train only a small number of additional parameters while keeping the pre-trained weights frozen. This insight, that task adaptation can be achieved through targeted modifications rather than wholesale changes, forms the conceptual foundation for all PEFT methods.

The PEFT Storage AdvantageLink Copied

The core principle behind PEFT's storage efficiency is strikingly simple. Rather than creating independent copies of the entire model for each task, PEFT stores a single copy of the base model and supplements it with small, task-specific adapter modules. These adapters typically modify less than 1% of total parameters, yet they capture the task-specific knowledge needed for strong performance. This architectural decision fundamentally changes the storage equation from a multiplicative relationship to an additive one.

To make the mathematics concrete: with full fine-tuning, total storage grows as:

S_{\text{full}} = (N_{\text{tasks}} + 1) \times S_{\text{model}}

where:

$N_{\text{tasks}}$ : the number of specialized tasks requiring a dedicated model variant
$S_{\text{model}}$ : the storage required for a single complete model
The " $+1$ " accounts for the base model that must also be retained

With PEFT, total storage follows a fundamentally different formula:

S_{\text{PEFT}} = S_{\text{model}} + N_{\text{tasks}} \times S_{\text{adapter}}

where:

$S_{\text{model}}$ : the storage for the single shared base model (same as before, but stored only once)
$S_{\text{adapter}}$ : the storage for each task-specific adapter, typically $\approx 0.01 \times S_{\text{model}}$

The key insight is that in the PEFT formula, the base model cost $S_{\text{model}}$ is a constant, not a multiplier. Adding a new task costs only $S_{\text{adapter}}$ , which is roughly 1% of $S_{\text{model}}$ . As you add more tasks, the total PEFT storage grows almost imperceptibly while the full fine-tuning storage grows by a full model copy each time. Let's see this principle in action:

In[16]:

Code

def calculate_peft_storage(
    base_model_params: int,
    num_tasks: int,
    peft_ratio: float = 0.01,  # Fraction of params per task
    precision: str = "fp16",
) -> dict:
    """Calculate storage with PEFT approach."""

    bytes_per_param = {"fp32": 4, "fp16": 2, "bf16": 2}[precision]

    # Base model: stored once
    base_model_bytes = base_model_params * bytes_per_param
    base_model_gb = base_model_bytes / (1024**3)

    # PEFT adapters: one small set per task
    adapter_params = base_model_params * peft_ratio
    adapter_bytes_per_task = adapter_params * bytes_per_param
    adapter_gb_per_task = adapter_bytes_per_task / (1024**3)

    total_adapter_gb = adapter_gb_per_task * num_tasks
    total_gb = base_model_gb + total_adapter_gb

    return {
        "base_model_gb": base_model_gb,
        "adapter_per_task_gb": adapter_gb_per_task,
        "adapter_per_task_mb": adapter_gb_per_task * 1024,
        "total_adapters_gb": total_adapter_gb,
        "total_storage_gb": total_gb,
        "num_adapter_params": adapter_params,
    }

In[17]:

Code

comparison_results = []

num_tasks = len(tasks)

for model_name, params in [
    ("LLaMA-7B", 7e9),
    ("LLaMA-13B", 13e9),
    ("LLaMA-70B", 70e9),
]:
    full_ft = calculate_deployment_storage(params, num_tasks, "fp16")
    peft = calculate_peft_storage(
        params, num_tasks, peft_ratio=0.01, precision="fp16"
    )
    savings_ratio = full_ft["total_storage_gb"] / peft["total_storage_gb"]
    comparison_results.append((model_name, full_ft, peft, savings_ratio))

Out[18]:

Console

PEFT vs Full Fine-tuning Storage Comparison
======================================================================
Scenario: 8 task-specific models, FP16 precision, 1% PEFT parameters

LLaMA-7B:
  Full fine-tuning: 117.3 GB (9 complete models)
  PEFT approach:    14.1 GB (1 base + 8 adapters @ 134 MB each)
  Storage savings:  8.3x reduction

LLaMA-13B:
  Full fine-tuning: 217.9 GB (9 complete models)
  PEFT approach:    26.2 GB (1 base + 8 adapters @ 248 MB each)
  Storage savings:  8.3x reduction

LLaMA-70B:
  Full fine-tuning: 1173.5 GB (9 complete models)
  PEFT approach:    140.8 GB (1 base + 8 adapters @ 1335 MB each)
  Storage savings:  8.3x reduction

The savings are dramatic. With PEFT, deploying eight LLaMA-70B variants requires 140 GB instead of 1,170 GB, over an 8x reduction. The base model is stored once, and each task adds only a small adapter. Notice how the savings ratio improves as we scale to larger models and more tasks, precisely because the adapter overhead becomes proportionally smaller relative to the base model. For organizations deploying dozens of specialized models, these savings translate directly into reduced infrastructure costs and simplified operations.

Out[19]:

Visualization

Line chart comparing total storage in GB versus number of tasks for full fine-tuning and PEFT, with savings region shaded green. — Storage scaling comparison as the number of tasks increases for LLaMA-70B in FP16. Full fine-tuning grows linearly, adding 130 GB per task. PEFT grows almost imperceptibly, adding only 1.3 GB per adapter. The shaded green region represents cumulative storage savings, which compound rapidly as task count grows.

Visualizing the Storage BreakdownLink Copied

Out[20]:

Visualization

Stacked bar chart showing nine equal-sized model copies totalling over 1,150 GB for full fine-tuning of LLaMA-70B. — Storage footprint for full fine-tuning across 8 tasks. Each task demands a complete independent copy of the 70B model in FP16, resulting in nine equally-sized stacked bars and a total storage exceeding 1,150 GB.

Stacked bar chart showing one large base model with eight tiny adapter bars on top, totalling approximately 140 GB, with an inset showing adapter sizes. — Storage footprint for PEFT across 8 tasks. The base model is stored once as the dominant block, and eight task-specific adapters appear as a barely-visible stack on top. The inset zooms in on the adapter sizes to reveal that each adapter occupies only about 1.3 GB, roughly 1% of the base model size.

The visualization illustrates how the base model dominates the storage footprint in PEFT, whereas full fine-tuning replicates the massive model for every task. The inset panel zooms in on the adapter sizes, revealing that all eight task-specific adapters combined occupy less space than a single percentage point of the base model. This contrast shows the scalability advantage of the PEFT approach.

What the visualization also reveals is an important property of the PEFT storage structure: the base model becomes a shared infrastructure asset. All tasks benefit from improvements to the base model, and upgrading the base model automatically upgrades the capability foundation for every downstream task. With full fine-tuning, upgrading the base model requires retraining all task-specific variants from scratch. With PEFT, you simply keep your adapters and load them on top of the new base model, potentially with just a brief round of re-tuning to re-align the adapter with the new weights.

Training Efficiency GainsLink Copied

PEFT reduces training costs beyond just storage. With fewer trainable parameters, you need less GPU memory because gradients and optimizer states only accumulate for the parameters that require updates. The frozen base model participates in the forward pass to compute activations, but it does not need gradient storage or optimizer state tracking. This selective allocation of training resources produces substantial memory savings:

In[21]:

Code

def calculate_peft_training_memory(
    num_params: int, peft_ratio: float = 0.01, precision: str = "fp16"
) -> dict:
    """
    Calculate training memory with PEFT.

    Key insight: Only PEFT parameters need gradients and optimizer states.
    The frozen base model only needs forward pass memory.
    """

    bytes_per_param = {"fp32": 4, "fp16": 2, "bf16": 2}[precision]

    trainable_params = num_params * peft_ratio
    frozen_params = num_params * (1 - peft_ratio)

    # Frozen parameters: only need to store weights
    frozen_memory = frozen_params * bytes_per_param

    # Trainable parameters: weights + gradients + optimizer states
    trainable_weights = trainable_params * bytes_per_param
    trainable_gradients = trainable_params * bytes_per_param

    # AdamW optimizer states (FP32)
    optimizer_states = trainable_params * 4 * 2  # m and v buffers
    if precision in ["fp16", "bf16"]:
        optimizer_states += trainable_params * 4  # FP32 master weights

    total_memory = (
        frozen_memory
        + trainable_weights
        + trainable_gradients
        + optimizer_states
    )

    return {
        "frozen_params_gb": frozen_memory / (1024**3),
        "trainable_weights_gb": trainable_weights / (1024**3),
        "trainable_gradients_gb": trainable_gradients / (1024**3),
        "optimizer_states_gb": optimizer_states / (1024**3),
        "total_gb": total_memory / (1024**3),
        "trainable_params": trainable_params,
    }

Notice how this function distinguishes between frozen and trainable parameters. The frozen parameters, comprising 99% of the model, require only storage for their weights. They flow through the forward computation but contribute nothing to the backward pass's memory footprint. In contrast, the trainable parameters, though representing only 1% of the total, carry the full burden of gradients and optimizer states. This asymmetric treatment is what enables the dramatic memory reduction.

To understand this more concretely: during backpropagation, PyTorch (or any modern deep learning framework) computes gradients only for parameters where requires_grad=True. For PEFT, we freeze the base model parameters by setting requires_grad=False, which tells the framework to skip gradient computation and storage for those weights. The computation graph still passes through the frozen layers during the forward pass, but the backward pass does not need to accumulate gradient tensors for them. This surgical separation between "frozen infrastructure" and "trainable adaptation" is what makes the memory reduction possible.

In[22]:

Code

# Calculate comparison for LLaMA-70B
full_ft_train = calculate_training_memory(70e9, "fp16", "adamw")
peft_train = calculate_peft_training_memory(
    70e9, peft_ratio=0.01, precision="fp16"
)

savings = full_ft_train["total_gb"] / peft_train["total_gb"]
full_gpus = int(np.ceil(full_ft_train["total_gb"] / 80))
peft_gpus = int(np.ceil(peft_train["total_gb"] / 80))

Out[23]:

Console

Training Memory: Full Fine-tuning vs PEFT (LLaMA-70B)
======================================================================

Full Fine-tuning (all 70B parameters trainable):
  Model parameters:  130.4 GB
  Gradients:         130.4 GB
  Optimizer states:  782.3 GB
  Total:             1043.1 GB

PEFT (1% = 700M parameters trainable):
  Frozen parameters: 129.1 GB
  Trainable weights: 1.30 GB
  Gradients:         1.30 GB
  Optimizer states:  7.8 GB
  Total:             139.5 GB

Memory reduction: 7.5x fewer GB required
A100-80GB GPUs needed: 14 (full) vs 2 (PEFT)

Out[24]:

Visualization

Stacked bar chart for full fine-tuning showing parameters, gradients, and optimizer states totalling over 1,000 GB. — Memory breakdown for full fine-tuning of LLaMA-70B in FP16 with AdamW. All 70 billion parameters require gradient and optimizer state tracking, pushing total training memory above 1,000 GB and necessitating at least 14 A100-80GB GPUs.

Stacked bar chart for PEFT showing frozen parameters as the dominant block with small trainable, gradient, and optimizer bars, totalling around 140 GB. — Memory breakdown for PEFT fine-tuning of LLaMA-70B with 1% trainable parameters. Gradient and optimizer state overhead applies only to the 700M adapter parameters, reducing total training memory to roughly 140 GB and making two A100-80GB GPUs sufficient.

PEFT reduces training memory by roughly 7x for LLaMA-70B, dropping from over 1,000 GB to approximately 140 GB. More importantly, this shifts training from requiring 14 or more A100 GPUs to potentially fitting on just 2 GPUs. Research labs and startups, as well as individual practitioners can fine-tune state-of-the-art models on accessible hardware. This enables a much broader community to adapt these systems to specialized tasks.

The accessibility change affects who can conduct this research. Before PEFT, adapter research was largely confined to well-funded industrial labs that could afford the compute. PEFT methods moved meaningful fine-tuning capability onto hardware available at universities, small companies, and even individual researchers with a single high-end workstation. This shift accelerated applied NLP research and produced specialized models that would have been economically infeasible to create just a few years earlier.

Adapter Swapping During InferenceLink Copied

Beyond the training benefits, PEFT enables a powerful deployment pattern: hot-swapping adapters without reloading the base model. In traditional full fine-tuning deployments, switching between tasks requires unloading one multi-gigabyte model from GPU memory and loading another in its place. This process takes 30-60 seconds for large models, creating unacceptable latency for interactive applications. PEFT fundamentally changes this dynamic by decoupling the base model from the task-specific components.

Think of adapter swapping as the software equivalent of hot-plugging USB devices. The computer, analogous to the base model, stays on and running. You simply plug in a different peripheral, analogous to a different adapter, and the system reconfigures in milliseconds. The underlying infrastructure never restarts; only the task-specific interface changes.

Out[25]:

Visualization

Diagram showing a central base model with four task-specific adapters connected by arrows, with one adapter highlighted as active. — PEFT multi-task serving architecture showing adapter hot-swapping. The base LLaMA-70B model remains loaded in GPU memory at all times, while small task-specific adapters (shown in colored boxes) can be swapped in milliseconds. The green dashed border marks the currently active adapter. This design eliminates the 30-60 second model reload latency of full fine-tuning deployments.

With adapter swapping, task switching takes milliseconds instead of the 30-60 seconds required to reload full models. The base model occupies GPU memory once, and small adapters load nearly instantaneously. This architecture enables new deployment patterns: a single GPU can serve multiple applications by keeping adapters ready in system memory and swapping them into the active computation path as requests arrive. The result is better hardware utilization, lower costs, and faster response times for multi-task deployments.

The adapter swapping pattern also enables something that was previously impractical: personalized model variants. Imagine an organization that serves different users or customer segments, each with slightly different style preferences, domain vocabularies, or safety requirements. With full fine-tuning, maintaining per-user or per-segment model variants is economically impossible. With PEFT adapters, you could maintain thousands of tiny adapter files, each representing a personalized configuration, while sharing a single base model instance in hardware.