Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

IA3 › IA3 with PEFT Library

Michael BrenndoerferDecember 6, 20254 min read

Language AI HandbookAll chapters

Part 35 of 67Parameter-Efficient Fine-tuning

Chapter 261IA3 › IA3 with PEFT Library

Reading section 8 of 11 · IA3 with PEFT Library · 4 minSection 8 of 11 · 4 min

Section 8 of 11 · 4 min read

IA3 › IA3 with PEFT LibraryOne section · Sections in this chapter8 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The Hugging Face PEFT library gives a production-quality implementation of IA3 that integrates smoothly with transformers models. This saves you from implementing the rescaling mechanism yourself and handles all the bookkeeping of which modules to adapt, how to save and load adapters, and how to merge adapters for inference.

Understanding how to configure IA3 through PEFT requires knowing the module names for the target components in your specific model architecture. Different transformer implementations use different naming conventions for their key, value, and feed-forward modules. You will need to inspect your model's named modules to identify the correct names. For DistilBERT (a compact BERT variant), the attention key and value projections are named k_lin and v_lin, and the first feed-forward layer is named lin1.

In[25]:

Code

from peft import IA3Config, TaskType, get_peft_model
from transformers import AutoConfig, AutoModelForSequenceClassification

# Load a base model setup (avoids downloading large weights)
model_name = "distilbert-base-uncased"
config = AutoConfig.from_pretrained(model_name, num_labels=2)
model = AutoModelForSequenceClassification.from_config(config)

# Configure IA3 (feedforward_modules must be subset of target_modules)
ia3_config = IA3Config(
    task_type=TaskType.SEQ_CLS,
    target_modules=["k_lin", "v_lin", "lin1"],
    feedforward_modules=["lin1"],
    modules_to_save=["classifier"],
    init_ia3_weights=True,
)

# Apply IA3 to the model
ia3_model = get_peft_model(model, ia3_config)

Out[26]:

Console

trainable params: 605,954 || all params: 67,560,964 || trainable%: 0.8969

The IA3Config parameters deserve careful attention. The target_modules list specifies all modules that will receive IA3 rescaling vectors. The feedforward_modules list must be a subset of target_modules and identifies which modules are feed-forward layers (as opposed to attention key/value projections). This distinction matters because the PEFT library needs to know the type of each adapted module to apply the rescaling at the correct point in the computation. The modules_to_save list identifies modules that should be fully unfrozen and trained, not adapted with IA3. Here, the classification head (classifier) needs to be fully trained since it is randomly initialized and needs to learn its task from scratch. The init_ia3_weights=True flag ensures that rescaling vectors are initialized to ones, preserving the pretrained model's behavior at training start.

In[27]:

Code

# Examine the IA3 parameters
ia3_params_dict = {
    name: param.shape
    for name, param in ia3_model.named_parameters()
    if param.requires_grad
}

Out[28]:

Console

Trainable IA3 Parameters:
--------------------------------------------------
base_model.model.distilbert.transformer.layer.0.attention.k_lin.ia3_l.default: torch.Size([768, 1])
base_model.model.distilbert.transformer.layer.0.attention.v_lin.ia3_l.default: torch.Size([768, 1])
base_model.model.distilbert.transformer.layer.0.ffn.lin1.ia3_l.default: torch.Size([1, 768])
base_model.model.distilbert.transformer.layer.1.attention.k_lin.ia3_l.default: torch.Size([768, 1])
base_model.model.distilbert.transformer.layer.1.attention.v_lin.ia3_l.default: torch.Size([768, 1])
base_model.model.distilbert.transformer.layer.1.ffn.lin1.ia3_l.default: torch.Size([1, 768])
base_model.model.distilbert.transformer.layer.2.attention.k_lin.ia3_l.default: torch.Size([768, 1])
base_model.model.distilbert.transformer.layer.2.attention.v_lin.ia3_l.default: torch.Size([768, 1])
base_model.model.distilbert.transformer.layer.2.ffn.lin1.ia3_l.default: torch.Size([1, 768])
base_model.model.distilbert.transformer.layer.3.attention.k_lin.ia3_l.default: torch.Size([768, 1])
... and 12 more parameters

Each adapted layer receives its own set of rescaling vectors. The IA3 vectors are named with ia3_l in their parameter names, making them easy to identify programmatically. The PEFT library stores these vectors efficiently and gives utilities for saving only the adapter weights (a tiny fraction of the full model size), loading adapters onto different base models, and merging adapters for zero-overhead deployment.

Saving and Loading IA3 AdaptersLink Copied

One of the most important practical advantages of IA3 is the tiny size of the saved adapter. Since only the rescaling vectors are saved, an IA3 adapter for a large model can be just a few kilobytes:

In[8]:

Code

# Save only the IA3 adapter (not the full model)
ia3_model.save_pretrained("./ia3_adapter/")

# Later: load the base model and apply the adapter
from peft import PeftModel
from transformers import AutoModelForSequenceClassification

base_model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=2
)
loaded_model = PeftModel.from_pretrained(base_model, "./ia3_adapter/")

The saved adapter directory contains only the IA3 rescaling vectors, not the base model weights. This means you can store hundreds of task-specific adapters in the space required for a single copy of the base model. At inference time, you load the base model once and hot-swap the adapter vectors to switch between tasks. This architecture is useful for serving many specialized language model capabilities in production environments with limited GPU memory.