PEFT Motivation › Worked Example: Calculating PEFT Savings

Michael BrenndoerferNovember 29, 20252 min read
Language AI HandbookAll chapters
Part 35 of 67Parameter-Efficient Fine-tuning
Chapter 254PEFT Motivation › Worked Example: Calculating PEFT Savings
Reading section 5 of 10 · Worked Example: Calculating PEFT Savings · 2 minSection 5 of 10 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 254 of 513

Section 5 of 10 · 2 min read

PEFT Motivation › Worked Example: Calculating PEFT SavingsOne section · Sections in this chapter5 / 10 ↓
Sections in this chapter10
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's work through a concrete numerical example to make the storage and training memory savings tangible. Consider a mid-sized company that wants to deploy a code assistant, a customer support bot, a legal document analyzer, and a medical question-answering system, each based on LLaMA-13B in FP16.

Step 1: Establish the baseline.

LLaMA-13B has 13 billion parameters. In FP16, each parameter requires 2 bytes:

Smodel=13×109×2 bytes=26×109 bytes≈24.2 GBS_{\text{model}} = 13 \times 10^9 \times 2 \text{ bytes} = 26 \times 10^9 \text{ bytes} \approx 24.2 \text{ GB}

Step 2: Calculate full fine-tuning storage for 4 tasks.

With full fine-tuning, we need the base model plus one fine-tuned copy per task:

Sfull=(4+1)×24.2 GB=121 GBS_{\text{full}} = (4 + 1) \times 24.2 \text{ GB} = 121 \text{ GB}

Step 3: Calculate PEFT storage for 4 tasks.

With 1% PEFT parameters, each adapter contains 13×109×0.01=13013 \times 10^9 \times 0.01 = 130 million trainable parameters. Each adapter in FP16 requires:

Sadapter=130×106×2 bytes=260×106 bytes≈0.24 GB=242 MBS_{\text{adapter}} = 130 \times 10^6 \times 2 \text{ bytes} = 260 \times 10^6 \text{ bytes} \approx 0.24 \text{ GB} = 242 \text{ MB}

Total PEFT storage for 4 tasks:

SPEFT=24.2 GB+4×0.24 GB=24.2+0.97≈25.2 GBS_{\text{PEFT}} = 24.2 \text{ GB} + 4 \times 0.24 \text{ GB} = 24.2 + 0.97 \approx 25.2 \text{ GB}

Step 4: Compute the storage reduction factor.

Reduction=SfullSPEFT=12125.2≈4.8x\text{Reduction} = \frac{S_{\text{full}}}{S_{\text{PEFT}}} = \frac{121}{25.2} \approx 4.8\text{x}

Step 5: Compare training memory.

Full fine-tuning of LLaMA-13B in FP16 with AdamW requires:

  • Parameters: 24.2 GB
  • Gradients: 24.2 GB
  • Optimizer states (2 FP32 buffers + FP32 master): 13×109×12 bytes≈145.513 \times 10^9 \times 12 \text{ bytes} \approx 145.5 GB
  • Total: approximately 194 GB, requiring 3 A100-80GB GPUs

PEFT training requires:

  • Frozen parameters: 24.2 GB (no gradients needed)
  • Trainable weights: 0.24 GB
  • Trainable gradients: 0.24 GB
  • Optimizer states for 130M trainable params: 130×106×12≈1.46130 \times 10^6 \times 12 \approx 1.46 GB
  • Total: approximately 26.1 GB, fitting comfortably on a single A100-40GB GPU

This is the power of PEFT in practice: a task that previously required 3 expensive GPUs and 194 GB of memory can now run on a single consumer-grade GPU with 26 GB of memory.