LoRA Implementation › HuggingFace PEFT Library
Language AI HandbookAll chapters LoRA Implementation › HuggingFace PEFT LibraryOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
While understanding the implementation details is useful for building intuition and debugging issues, production systems typically use the PEFT (Parameter-Efficient Fine-Tuning) library from HuggingFace. PEFT gives optimized implementations of LoRA and other adaptation methods that integrate with the transformers ecosystem. Using PEFT means you benefit from tested, maintained code that handles edge cases and optimizations you might miss in a custom implementation.
The PEFT library handles several complexities that our custom implementation glosses over. It correctly handles layers with tied weights, modules that appear multiple times in the computation graph, and the interaction between LoRA and model parallel configurations used for very large models. It also integrates with HuggingFace's Trainer class, meaning you can use all the standard training utilities like gradient accumulation, mixed precision, and learning rate scheduling without any special modifications for LoRA.
Let's walk through a practical example of fine-tuning a language model with PEFT:
# Import required packages (assuming they are already installed)
# If not installed, run: pip install peft transformersfrom transformers import DistilBertConfig, DistilBertForSequenceClassification
# In a real scenario, you would download pre-trained weights:
# model_name = "distilbert-base-uncased"
# tokenizer = AutoTokenizer.from_pretrained(model_name)
# base_model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
# For this demonstration, we initialize a random model to avoid downloads
config = DistilBertConfig(
n_layers=2,
n_heads=4,
dim=256,
hidden_dim=1024,
vocab_size=2000,
num_labels=2,
)
base_model = DistilBertForSequenceClassification(config)
# Create a dummy tokenizer for demonstration
class DummyTokenizer:
def __call__(
self,
texts,
truncation=True,
padding="max_length",
max_length=128,
return_tensors="pt",
):
batch_size = len(texts)
return {
"input_ids": torch.randint(0, 2000, (batch_size, max_length)),
"attention_mask": torch.ones(
batch_size, max_length, dtype=torch.long
),
}
tokenizer = DummyTokenizer()The PEFT library uses a configuration object to specify LoRA hyperparameters. This gives a clean interface for customizing the adaptation behavior. The configuration object separates the concerns of "what LoRA should do" from "how to apply it to a model," which makes it easy to swap configurations without touching the model-wrapping code:
from peft import LoraConfig, TaskType, get_peft_model
# Configure LoRA
lora_config = LoraConfig(
task_type=TaskType.SEQ_CLS,
r=8,
lora_alpha=16,
lora_dropout=0.1,
target_modules=["q_lin", "v_lin"],
bias="none",
modules_to_save=None,
)
# Wrap the model with LoRA
peft_model = get_peft_model(base_model, lora_config)
# Get parameter statistics
trainable_params, all_params = peft_model.get_nb_trainable_parameters()trainable params: 82,690 || all params: 2,372,100 || trainable%: 3.4859
The output shows the large reduction in trainable parameters achieved by PEFT. With just a single function call, the library has identified the target modules, wrapped them with LoRA adapters, and frozen the base weights. The bias="none" parameter tells PEFT not to train any bias terms alongside the LoRA matrices, which is the most memory-efficient option. Alternatively, you can set bias="all" to train all biases or bias="lora_only" to train only the biases in LoRA-targeted layers.
Let's examine the modified architecture to understand what changed:
# Examine the LoRA structure in one attention layer
target_name = None
target_module = None
for name, module in peft_model.named_modules():
if "lora" in name.lower() and "q_lin" in name:
target_name = name
target_module = module
breakSample LoRA module structure: Module: base_model.model.distilbert.transformer.layer.0.attention.q_lin.lora_dropout Type: ModuleDict
The output confirms that the targeted linear layers have been successfully wrapped with LoRA adapters. These adapters intercept the forward pass, adding the low-rank adaptation path alongside the frozen base computation. The PEFT library uses its own Linear class internally, which is similar in structure to our LoRALinear but with additional features like support for half-precision training and gradient checkpointing.
Training with PEFT
Training a PEFT model is identical to training any HuggingFace model. The library handles freezing base weights and updating only LoRA parameters transparently, so your training code requires no special modifications. The fact that LoRA is present is invisible to the training loop, which is precisely the design goal: you should be able to drop LoRA into an existing training pipeline with minimal code changes.
from torch.utils.data import Dataset
class SimpleTextDataset(Dataset):
"""Minimal dataset for demonstration."""
def __init__(self, texts, labels, tokenizer, max_length=128):
self.encodings = tokenizer(
texts,
truncation=True,
padding="max_length",
max_length=max_length,
return_tensors="pt",
)
self.labels = torch.tensor(labels)
def __len__(self):
return len(self.labels)
def __getitem__(self, idx):
return {
"input_ids": self.encodings["input_ids"][idx],
"attention_mask": self.encodings["attention_mask"][idx],
"labels": self.labels[idx],
}
# Create sample data
texts = [
"This movie was absolutely wonderful!",
"Terrible film, waste of time.",
"A masterpiece of modern cinema.",
"Boring and predictable plot.",
"Outstanding performances by the cast!",
"I want my money back.",
"Beautiful cinematography throughout.",
"The worst movie I've ever seen.",
] * 10 # Repeat for more samples
labels = [1, 0, 1, 0, 1, 0, 1, 0] * 10
train_dataset = SimpleTextDataset(texts, labels, tokenizer)# Training loop
peft_model.train()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
peft_model = peft_model.to(device)
# Create optimizer with only trainable (LoRA) parameters
peft_optimizer = torch.optim.AdamW(
filter(lambda p: p.requires_grad, peft_model.parameters()),
lr=2e-4,
weight_decay=0.01,
)
train_dataloader = DataLoader(train_dataset, batch_size=8, shuffle=True)
peft_losses = []
for epoch in range(3):
epoch_loss = 0
for batch in train_dataloader:
batch = {k: v.to(device) for k, v in batch.items()}
outputs = peft_model(**batch)
loss = outputs.loss
peft_optimizer.zero_grad()
loss.backward()
peft_optimizer.step()
epoch_loss += loss.item()
avg_loss = epoch_loss / len(train_dataloader)
peft_losses.append(avg_loss)PEFT Training Progress: Epoch 1: Loss = 0.7042 Epoch 2: Loss = 0.6853 Epoch 3: Loss = 0.6760
The steady decrease in loss shows the model is learning from the dataset. Optimizing just the small set of LoRA parameters is sufficient to drive this learning, which validates the core hypothesis behind LoRA: that task-specific adaptations can be captured in a low-rank subspace of the full weight space. The model is not simply memorizing the training examples, it is discovering a compact parameterization of what "positive sentiment" and "negative sentiment" mean in terms of attention patterns.
Saving and Loading PEFT Models
PEFT gives convenient methods for saving only the adapter weights, which makes it easy to distribute lightweight task-specific adaptations. The library follows a consistent convention for file naming and format, which means adapters saved by one version of PEFT can generally be loaded by another, easing version management.
with tempfile.TemporaryDirectory() as tmpdir:
# Save only the LoRA adapter (not the full model)
peft_model.save_pretrained(tmpdir)
# List saved files
saved_files = os.listdir(tmpdir)
file_sizes = {
f: os.path.getsize(os.path.join(tmpdir, f)) for f in saved_files
}Saved adapter files: adapter_model.safetensors: 324.52 KB README.md: 5.01 KB adapter_config.json: 1.00 KB
The adapter checkpoint is just a few hundred kilobytes, compared to hundreds of megabytes for the full DistilBERT model. This makes possible practical workflows where you distribute a single base model with many task-specific adapters, letting you to quickly switch between different capabilities without downloading or storing multiple complete models. In the open-source ecosystem, adapters can be shared as standalone files on the HuggingFace Hub. Anyone with the base model can then apply your adapter with from_pretrained, creating a collaborative environment for model customization.
The adapter_config.json file saved alongside the weights is equally important. It records the LoRA configuration, including rank, alpha, target modules, and other settings. When loading the adapter later, PEFT reads this configuration to reconstruct the correct LoRA structure before loading the saved weights. This means you don't need to remember which configuration you used when training: the adapter is self-describing.
Merging and Unloading
PEFT also supports merging LoRA weights for inference. This gives a one-line solution for converting your adapted model into a standard format. The merge_and_unload method performs both operations: it merges all LoRA weights into the base model and removes the LoRA modules from the architecture, returning a clean model with no PEFT overhead:
# Merge LoRA weights into the base model
merged_model = peft_model.merge_and_unload()Merged model type: DistilBertForSequenceClassification Parameters: 2,289,410
After merging, you have a standard transformer model with no LoRA overhead, suitable for optimized inference pipelines. The merged model can be deployed using any existing serving infrastructure without special handling for LoRA adapters. This is particularly important for production deployments that use inference optimization techniques like TensorRT or ONNX export, which work best on standard model architectures without custom modules.