PEFT Motivation › The Parameter Storage Problem
Language AI HandbookAll chapters PEFT Motivation › The Parameter Storage ProblemOne section · Sections in this chapter2 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Modern language models contain billions of parameters, each stored as a floating-point number. Every weight in a neural network occupies physical memory on a GPU or in system RAM. When you multiply billions of parameters by the bytes required for each floating-point representation, the resulting storage requirements quickly reach sizes that strain even enterprise-grade hardware. Understanding these raw storage costs reveals why full fine-tuning doesn't scale to practical deployment scenarios. Before we can appreciate the need for parameter efficiency, we need to hold the actual numbers in our heads.
The key insight is that storage costs are not an engineering inconvenience you can eventually optimize away. They are a direct consequence of model scale, and model scale is what delivers capability. You cannot have the language understanding power of a 70B model without storing those 70 billion weights somewhere. The question is not whether to pay the storage cost for the base model, but whether you must pay it again, multiplied by the number of tasks you need to handle.
Think of the parameter storage problem as the data center equivalent of a library. A single copy of every book takes a certain amount of shelf space. If you decide that every reader needs their own personal library with slight annotations in the margins, you immediately need thousands of times more shelf space, even though most of the text is identical. PEFT is the library version of asking readers to keep just their personal margin notes and share the original books.
Quantifying Model Sizes
Before we can appreciate the need for parameter efficiency, we must first establish a clear picture of how much space these models consume. The storage footprint depends on two factors: the number of parameters in the model and the numerical precision used to represent each parameter. Common precision formats include FP32 (32-bit floating point, requiring 4 bytes per parameter), FP16 and BF16 (16-bit formats, requiring 2 bytes each), and quantized formats like INT8 (1 byte) or INT4 (half a byte). Let's calculate the storage requirements for popular LLM architectures across these different precision levels:
def calculate_model_size(num_params: int, precision: str = "fp32") -> dict:
"""Calculate storage requirements for a model."""
bytes_per_param = {
"fp32": 4, # 32-bit floating point
"fp16": 2, # 16-bit floating point
"bf16": 2, # Brain floating point
"int8": 1, # 8-bit integer (quantized)
"int4": 0.5, # 4-bit integer (quantized)
}
bytes_needed = num_params * bytes_per_param[precision]
return {
"params_billions": num_params / 1e9,
"bytes": bytes_needed,
"gigabytes": bytes_needed / (1024**3),
"precision": precision,
}
# Popular model sizes (parameter counts)
models = {
"BERT-base": 110e6,
"BERT-large": 340e6,
"GPT-2": 1.5e9,
"LLaMA-7B": 7e9,
"LLaMA-13B": 13e9,
"LLaMA-70B": 70e9,
"Mixtral-8x7B": 46.7e9, # Total params (not active)
"GPT-3": 175e9,
}import pandas as pd
# Calculate sizes for different precisions
results = []
for model_name, params in models.items():
for precision in ["fp32", "fp16", "int8"]:
size = calculate_model_size(params, precision)
results.append(
{
"Model": model_name,
"Parameters": f"{size['params_billions']:.1f}B",
"Precision": precision.upper(),
"Size (GB)": f"{size['gigabytes']:.1f}",
}
)
df = pd.DataFrame(results)
pivot_df = df.pivot(index="Model", columns="Precision", values="Size (GB)")
pivot_df = pivot_df[["FP32", "FP16", "INT8"]]
# Sort by model size
model_order = [
"BERT-base",
"BERT-large",
"GPT-2",
"LLaMA-7B",
"LLaMA-13B",
"Mixtral-8x7B",
"LLaMA-70B",
"GPT-3",
]
pivot_df = pivot_df.reindex(model_order)Model Storage Requirements by Precision ================================================== Precision FP32 FP16 INT8 Model BERT-base 0.4 0.2 0.1 BERT-large 1.3 0.6 0.3 GPT-2 5.6 2.8 1.4 LLaMA-7B 26.1 13.0 6.5 LLaMA-13B 48.4 24.2 12.1 Mixtral-8x7B 174.0 87.0 43.5 LLaMA-70B 260.8 130.4 65.2 GPT-3 651.9 326.0 163.0
The numbers illustrate the challenge clearly. A single LLaMA-70B model requires 130 GB in FP16 precision, which is the standard format for fine-tuning since it balances numerical stability with memory efficiency. Even with aggressive INT8 quantization, you still need 65 GB just for model weights. To put this in perspective, a high-end consumer GPU like the NVIDIA RTX 4090 has only 24 GB of memory, meaning even quantized versions of these large models cannot fit on typical hardware. This storage requirement represents just the static weights, before we consider the additional memory needed during training or inference.
What is striking about the table above is how predictable the growth pattern is. FP16 storage in gigabytes equals the parameter count in billions multiplied by approximately 1.86. So a 7B model needs about 13 GB, a 13B model needs about 24 GB, a 70B model needs about 130 GB, and a 175B model like GPT-3 needs about 325 GB. These are not edge cases. These are the baseline storage requirements before you do anything with the model.
The Full Fine-tuning Multiplication Problem
The storage challenge described above represents the cost for a single model, but real-world deployments rarely involve just one task. Organizations typically need models specialized for multiple applications: customer support chatbots, code assistants, document summarization tools, and more. With full fine-tuning, each task requires its own complete copy of the model, causing storage requirements to multiply linearly with the number of tasks. Consider a company deploying LLMs for several applications:
def calculate_deployment_storage(
base_model_params: int, num_tasks: int, precision: str = "fp16"
) -> dict:
"""Calculate total storage for multi-task deployment with full fine-tuning."""
single_model_size = calculate_model_size(base_model_params, precision)
# Full fine-tuning: complete copy for each task
total_storage_gb = single_model_size["gigabytes"] * num_tasks
# Plus the base model itself
total_with_base = total_storage_gb + single_model_size["gigabytes"]
return {
"single_model_gb": single_model_size["gigabytes"],
"num_models": num_tasks + 1, # fine-tuned + base
"total_storage_gb": total_with_base,
"total_storage_tb": total_with_base / 1024,
}
# Common deployment scenarios
tasks = [
"customer_support",
"code_generation",
"legal_analysis",
"medical_qa",
"translation",
"summarization",
"sentiment_analysis",
"named_entity_recognition",
]deployment_scenarios = []
for model_name, params in [
("LLaMA-7B", 7e9),
("LLaMA-13B", 13e9),
("LLaMA-70B", 70e9),
]:
result = calculate_deployment_storage(params, len(tasks), "fp16")
deployment_scenarios.append((model_name, result))Storage Requirements for Multi-Task Deployment (FP16) ============================================================ Tasks: 8 specialized models + 1 base model LLaMA-7B: Single model: 13.0 GB Total (9 models): 117.3 GB (0.11 TB) LLaMA-13B: Single model: 24.2 GB Total (9 models): 217.9 GB (0.21 TB) LLaMA-70B: Single model: 130.4 GB Total (9 models): 1173.5 GB (1.15 TB)
The multiplication effect becomes dramatic at scale. Deploying eight task-specific LLaMA-70B models requires over 1 TB of storage for weights alone. This calculation doesn't even include optimizer states, activation checkpoints, or the redundant copies needed for high availability in production systems. At the enterprise level, where organizations might have dozens of specialized applications, the storage infrastructure costs become prohibitive. This linear scaling with task count represents a fundamental limitation of the full fine-tuning paradigm.
The deeper problem is not merely storage capacity: it's the operational overhead. Each of those model copies must be version-controlled, backed up, potentially retrained when the base model improves, and served from appropriate hardware. When a new base model checkpoint becomes available, upgrading nine task-specific fine-tuned models requires nine separate training runs, nine rounds of evaluation, and nine new deployments. The engineering burden scales with the number of tasks in a way that quickly outpaces team capacity.
Training Memory Requirements
Storage is only part of the problem. While storage determines how many models you can save to disk, training memory determines whether you can update those models at all. The training process requires substantially more memory than inference because the GPU must simultaneously hold the model parameters, the gradients computed during backpropagation, and the optimizer states that track momentum and variance for each parameter. Each of these components consumes memory proportional to the parameter count:
def calculate_training_memory(
num_params: int, precision: str = "fp16", optimizer: str = "adamw"
) -> dict:
"""
Calculate GPU memory requirements for training.
Memory components:
- Model parameters
- Gradients (same size as parameters)
- Optimizer states (varies by optimizer)
- Activations (batch/sequence dependent, not calculated here)
"""
bytes_per_param = {"fp32": 4, "fp16": 2, "bf16": 2}[precision]
# Base parameter memory
param_memory = num_params * bytes_per_param
# Gradients (same precision as parameters)
gradient_memory = num_params * bytes_per_param
# Optimizer states
# AdamW stores: m (momentum), v (variance) - typically in FP32
if optimizer == "adamw":
optimizer_memory = num_params * 4 * 2 # Two FP32 buffers
# Also stores FP32 master weights if training in FP16
if precision in ["fp16", "bf16"]:
optimizer_memory += num_params * 4 # FP32 master weights
elif optimizer == "sgd_momentum":
optimizer_memory = num_params * 4 # One FP32 momentum buffer
else: # vanilla SGD
optimizer_memory = 0
total_memory = param_memory + gradient_memory + optimizer_memory
return {
"parameters_gb": param_memory / (1024**3),
"gradients_gb": gradient_memory / (1024**3),
"optimizer_gb": optimizer_memory / (1024**3),
"total_gb": total_memory / (1024**3),
}Let's examine how these components contribute to the total memory footprint. The model parameters themselves require storage proportional to their precision. Gradients, computed during the backward pass, require the same amount of memory since each parameter receives a corresponding gradient value. The optimizer states for AdamW are particularly memory-intensive because the algorithm maintains two running averages (momentum and variance) for each parameter, typically stored in full FP32 precision to ensure numerical stability. When training in mixed precision (FP16 or BF16), the optimizer also maintains a master copy of the weights in FP32 for accurate weight updates.
To see why this matters, consider the following: if you train in FP16 mixed precision with AdamW, each parameter effectively requires 2 bytes for the model weight, 2 bytes for the gradient, 4 bytes each for the momentum and variance buffers, and 4 bytes for the FP32 master weight. That is 16 bytes per parameter in total, compared to just 2 bytes for inference. Training uses eight times more memory per parameter than serving. For a 70B model, this translates from roughly 130 GB for inference to over 1,000 GB for training.
training_memory_results = []
for model_name, params in [
("LLaMA-7B", 7e9),
("LLaMA-13B", 13e9),
("LLaMA-70B", 70e9),
]:
mem = calculate_training_memory(params, "fp16", "adamw")
training_memory_results.append((model_name, mem))Training Memory Breakdown (FP16 mixed precision with AdamW) ================================================================= Model Params Gradients Optimizer Total ----------------------------------------------------------------- LLaMA-7B 13.0 13.0 78.2 104.3 LLaMA-13B 24.2 24.2 145.3 193.7 LLaMA-70B 130.4 130.4 782.3 1043.1 Note: This excludes activation memory, which depends on batch size and sequence length.
The memory requirements for training are substantial. Full fine-tuning of LLaMA-70B requires approximately 1,040 GB of GPU memory for parameters, gradients, and optimizer states alone. To understand what this means in practice, consider that a single A100 GPU, one of the most powerful GPUs available for machine learning, has 80 GB of memory. Simple arithmetic reveals that you need at least 14 A100 GPUs just for these components, before accounting for the activation memory required to store intermediate values during the forward pass. This hardware requirement places full fine-tuning of large models out of reach for most organizations, and entirely out of reach for individual researchers.
The cost is not merely academic. In cloud computing terms, a single A100 GPU rents for roughly $2-4 per hour depending on the provider and configuration. A 14-GPU setup for a single LLaMA-70B fine-tuning run costs $28-56 per hour. A training run that takes two days would cost $1,300-2,700 for the compute alone, before data preparation, iteration time, or human oversight. When you need to fine-tune eight task-specific variants, you are looking at potentially tens of thousands of dollars per training cycle, and that cost recurs every time you need to update the model.