Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

IA3 › Parameter Count Analysis

Michael BrenndoerferDecember 6, 20256 min read

Language AI HandbookAll chapters

Part 35 of 67Parameter-Efficient Fine-tuning

Chapter 261IA3 › Parameter Count Analysis

Reading section 4 of 11 · Parameter Count Analysis · 6 minSection 4 of 11 · 6 min

Section 4 of 11 · 6 min read

IA3 › Parameter Count AnalysisOne section · Sections in this chapter4 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding IA3's parameter count in exact quantitative terms is important for making informed decisions about when to use it versus alternatives. Let's derive the per-layer and total parameter counts carefully, then compare them to LoRA across representative model scales.

For a single standard transformer layer with hidden dimension $d$ and feed-forward dimension $d_{ff}$ (typically $4d$ ), IA3 introduces exactly three vectors. The total count per layer derives from summing these three vector lengths:

\begin{aligned} \text{IA3 params per layer} &= |\ell_K| + |\ell_V| + |\ell_{ff}| \\ &= d + d + d_{ff} \\ &= 2d + d_{ff} \end{aligned}

Substituting the typical $d_{ff} = 4d$ :

\begin{aligned} \text{IA3 params per layer} &= 2d + d_{ff} \\ &= 2d + 4d && \text{(substitute $d_{ff} = 4d$)} \\ &= 6d \end{aligned}

where:

$d$ : the hidden dimension of the model
$d_{ff}$ : the feed-forward dimension (typically $4d$ )
$2d$ : combined parameters for the key rescaling vector $\ell_K$ and value rescaling vector $\ell_V$
$4d$ : parameters for the feed-forward rescaling vector $\ell_{ff}$

This linear scaling with hidden dimension is gentle. As models grow from $d = 768$ (BERT-base) to $d = 4096$ (LLaMA-7B) to $d = 8192$ (larger frontier models), IA3's per-layer parameter count grows proportionally. Because the original model's weight matrices grow quadratically with $d$ (a $d \times d$ weight matrix has $d^2$ parameters), IA3's share of the total parameter count decreases as models get larger. IA3 therefore becomes relatively more efficient as models grow.

For LoRA adapting the same components (keys, values, and both feed-forward projections) with rank $r$ , the parameter count per layer is:

\begin{aligned} \text{LoRA params per layer} &= 2 \times 2dr + 2 \times (d + d_{ff})r \\ &= 4dr + 2(d + 4d)r && \text{(substitute $d_{ff} = 4d$)} \\ &= 4dr + 10dr \\ &= 14dr \end{aligned}

where:

$r$ : the LoRA rank
$d$ : the hidden dimension
$d_{ff}$ : the feed-forward dimension ( $4d$ )
$2 \times 2dr$ : the $A$ and $B$ matrices for both key and value projections (each projection needs an $r \times d$ matrix and a $d \times r$ matrix)
$2 \times (d + d_{ff})r$ : the $A$ and $B$ matrices for both feed-forward projections (mapping between $d$ and $d_{ff}$ dimensions)

The necessary difference is the factor of $r$ : LoRA's per-layer count grows linearly with rank. Even at $r = 1$ (the smallest possible rank), LoRA requires $14d$ parameters per layer versus IA3's $6d$ . At the commonly used rank $r = 8$ , LoRA needs $14 \times 8d = 112d$ parameters per layer, versus IA3's $6d$ , roughly 19 times more. At rank 16, the ratio grows to approximately 37 times more parameters.

For concrete numbers with LLaMA-7B ( $d = 4096$ , $d_{ff} = 11008$ , 32 layers) and rank $r = 8$ :

IA3 per layer: $2(4096) + 11008 = 19{,}200$ parameters
IA3 total (32 layers): $19{,}200 \times 32 = 614{,}400$ parameters
LoRA per layer (rank 8): $372{,}736$ parameters using the actual $d_{ff}=11{,}008$
LoRA total (32 layers): $11{,}927{,}552$ parameters
LoRA/IA3 ratio: approximately 19.4 times more parameters for LoRA

This compression, from about 11.9 million parameters for LoRA to 614,400 for IA3, is what makes IA3 stand apart. You could store about 19 IA3 task-specific adaptations in the memory required for a single rank-8 LoRA adaptation configured this way. For production systems serving many tasks, this density advantage can be decisive.

In[8]:

Code

def count_parameters(
    d_model: int, d_ff: int, num_layers: int, lora_rank: int = 8
):
    """Compare parameter counts for IA3 vs LoRA."""

    # IA3: rescaling vectors for keys, values, and feed-forward
    ia3_per_layer = 2 * d_model + d_ff  # l_k, l_v, l_ff
    ia3_total = ia3_per_layer * num_layers

    # LoRA: low-rank matrices for same components
    # Each adapted weight needs r*(d_in + d_out) parameters
    lora_k = lora_rank * (d_model + d_model)  # Key projection adaptation
    lora_v = lora_rank * (d_model + d_model)  # Value projection adaptation
    lora_ff1 = lora_rank * (d_model + d_ff)  # First FFN projection adaptation
    lora_ff2 = lora_rank * (d_ff + d_model)  # Second FFN projection adaptation
    lora_per_layer = lora_k + lora_v + lora_ff1 + lora_ff2
    lora_total = lora_per_layer * num_layers

    return {
        "ia3_per_layer": ia3_per_layer,
        "ia3_total": ia3_total,
        "lora_per_layer": lora_per_layer,
        "lora_total": lora_total,
        "ratio": lora_total / ia3_total,
    }


# LLaMA-7B setup
llama_7b = count_parameters(d_model=4096, d_ff=11008, num_layers=32)

# GPT-2 setup
gpt2 = count_parameters(d_model=768, d_ff=3072, num_layers=12)

Out[9]:

Console

Parameter Comparison: IA3 vs LoRA (rank=8)
==================================================

LLaMA-7B (4096 hidden, 32 layers):
  IA3 total:       614,400 params
  LoRA total:   11,927,552 params
  LoRA/IA3 ratio: 19.4x more params for LoRA

GPT-2 (768 hidden, 12 layers):
  IA3 total:        55,296 params
  LoRA total:    1,032,192 params
  LoRA/IA3 ratio: 18.7x more params for LoRA

Out[11]:

Visualization

Grouped bar chart with logarithmic y-axis comparing trainable parameters in millions for IA3, LoRA rank 8, and LoRA rank 16 across six model sizes from GPT-2 to LLaMA-13B. — Parameter scaling across six model sizes from GPT-2 (117M total params) to LLaMA-13B (13B total params). IA3 remains below 1M trainable parameters across these model sizes, while LoRA's count grows with both model width and rank. At rank 8, LoRA requires 18.7 to 19.4 times more parameters than IA3 for the configurations shown. At rank 16, the ratio is 37.3 to 38.8 times, so the absolute storage gap grows with model scale while the relative ratio stays nearly constant.

For LLaMA-7B, IA3 requires 614,400 trainable parameters, while LoRA with rank 8 on the same key, value, and feed-forward projections needs approximately 11.9 million. This 19.4-fold difference makes IA3 particularly attractive when memory is severely constrained or when you need to maintain and serve many task-specific adaptations simultaneously.