LoRA Hyperparameters › Rank Selection

The rank $r$ is LoRA's most important hyperparameter. It determines the dimensionality of the low-rank decomposition and directly controls the number of trainable parameters introduced. Understanding rank requires thinking carefully about what these low-rank matrices represent: they define a compressed subspace within which all adaptation must occur. A higher rank provides a richer, more flexible subspace capable of capturing more fine-grained transformations, while a lower rank constrains the adaptation to simpler patterns that can be expressed with fewer degrees of freedom.

Think of the rank as specifying the number of "directions" in which your adapter can modify the model's behavior. At rank 1, the entire adaptation is captured by a single direction in weight space: the adapter can only make transformations that correspond to scalar multiples of a single outer product. At rank 8, there are eight independent directions available. This allows the adapter to separately encode different aspects of the required transformation. At rank 64, there are 64 directions. This provides the capacity to represent quite complex modifications while still remaining far smaller than the full weight matrix.

The key insight is that for many fine-tuning tasks, the required transformation has low intrinsic dimensionality. Pre-trained models already learn rich, general representations. Task-specific adaptation primarily involves steering these existing capabilities rather than learning entirely new transformations. The model already knows how to process language; fine-tuning on a legal document classification task mostly requires the model to recognize which of its existing linguistic and semantic capabilities to prioritize for that domain. This steering operation turns out to require far fewer dimensions than the original weight matrices would suggest.

Recall from our discussion of LoRA mathematics that for a weight matrix $W \in \mathbb{R}^{d_{out} \times d_{in}}$, LoRA introduces:

$$\Delta W = BA$$

where:

• $\Delta W$: the low-rank update matrix that captures all task-specific modifications to the original weights
• $B$: the up-projection matrix ($\in \mathbb{R}^{d_{out} \times r}$), which maps from the compressed rank-$r$ space back to the output dimension
• $A$: the down-projection matrix ($\in \mathbb{R}^{r \times d_{in}}$), which projects the input into the compressed rank-$r$ space

The basic insight here is that the product $BA$ can only produce matrices of rank at most $r$, regardless of how large $d_{in}$ and $d_{out}$ might be. This means we are implicitly assuming that the difference between what the pre-trained model computes and what we need for our specific task can be well-approximated by a low-rank transformation. This assumption is surprisingly valid in practice, as we will see.

Since the adapter consists of matrices $A$ ($r \times d_{in}$) and $B$ ($d_{out} \times r$), the total parameter count is the sum of their elements. To understand this concretely, consider that matrix $A$ has $r$ rows and $d_{in}$ columns, yielding $r \times d_{in}$ individual learnable parameters. Similarly, matrix $B$ has $d_{out}$ rows and $r$ columns, contributing $d_{out} \times r$ parameters. The number of trainable parameters per adapted weight matrix is therefore:

$$\begin{aligned} \text{params} &= \underbrace{r \times d_{in}}_{\text{matrix } A} + \underbrace{d_{out} \times r}_{\text{matrix } B} \\ &= r \times (d_{in} + d_{out}) \end{aligned}$$

where:

• $\text{params}$: the total count of trainable parameters for the adapter
• $r$: the rank hyperparameter, the single value that controls the capacity of the entire adaptation
• $d_{out}$: the output dimension of the layer being adapted
• $d_{in}$: the input dimension of the layer being adapted

This formula reveals an elegant property: the parameter count scales linearly with rank. Doubling the rank exactly doubles the number of trainable parameters. It also scales linearly with the sum of dimensions rather than their product, which is the key insight that makes LoRA so efficient. For a square weight matrix where $d_{in} = d_{out} = d$, the original weight matrix has $d^2$ parameters while LoRA adds only $2rd$ parameters, a savings factor of $\frac{d}{2r}$. For a model with hidden dimension 4096 and rank 16, this savings factor is 128: the adapter uses roughly 0.78% of the parameters compared to storing the full weight update.

Let us examine how rank affects parameter counts for typical transformer dimensions. This analysis will make concrete the dramatic parameter savings that LoRA provides across different model scales.

The parameter efficiency is striking and reveals why LoRA has become so popular for resource-constrained fine-tuning. For a 4096-dimensional model (like LLaMA-7B's hidden dimension), rank 16 uses about 0.8% of the parameters compared to full fine-tuning of a single weight matrix. This represents a reduction of more than 100x in the parameters that must be stored and optimized, including those distributed across training runs. This efficiency is why LoRA enables fine-tuning on consumer hardware: where full fine-tuning of a 7B parameter model might require multiple high-end GPUs, LoRA adaptation can often be accomplished on a single GPU with 16GB or even less memory.

The x-axis is logarithmic, so each step doubles rank and doubles the LoRA parameter share. For a square $d \times d$ weight matrix, that share is $2r/d$: at the same rank, a larger hidden dimension yields a smaller adapter-to-matrix ratio. The adapter itself is still larger in absolute terms for the larger model because its parameter count grows with $d$.

Research and practitioner experience have converged on general guidelines that provide useful starting points. These recommendations emerge from extensive experimentation across diverse tasks and model architectures. They should be treated as starting points rather than rigid rules, since your specific task and data distribution may justify different choices.

• Rank 4-8: Suitable for simple classification tasks, sentiment analysis, and tasks where the pre-trained model already performs reasonably well. At these ranks, you are making only subtle adjustments to the model's behavior, which is often sufficient when the pre-training distribution closely matches your target domain. For binary sentiment classification of English product reviews, a model pre-trained on large English corpora already understands the language and sentiment concepts; it just needs minor steering.

• Rank 16-32: The sweet spot for most instruction tuning and domain adaptation tasks. These ranks provide enough capacity to learn new behavioral patterns while remaining extremely parameter-efficient. A rank-16 adapter on a 7B model has roughly 4 million trainable parameters, yet this is sufficient to teach the model to follow complex instruction formats, adopt specific output styles, or adapt to a particular domain's vocabulary.

• Rank 64-128: Appropriate for complex tasks requiring significant behavior changes, multilingual adaptation, or when data is abundant. Here, you have capacity comparable to more substantial model modifications. Cross-lingual transfer, for example, may require the adapter to encode a richer set of linguistic correspondences.

• Rank 256+: Rarely necessary; consider whether full fine-tuning might be more appropriate. At these ranks, you are approaching the parameter counts where the low-rank assumption provides diminishing benefits, and the computational savings become smaller relative to the overhead of managing the LoRA infrastructure.

The original LoRA paper found that even rank 4 achieved competitive results on many tasks, suggesting that the update matrices for adapting pre-trained models often have low intrinsic dimensionality. This observation aligns with findings from the intrinsic dimensionality literature, which showed that fine-tuning can be effective in surprisingly low-dimensional subspaces. The implication is that pre-trained models have already learned rich, general representations, and task-specific adaptation primarily involves steering these existing capabilities rather than learning entirely new transformations.

The relationship between task complexity and optimal rank is not strictly linear, and understanding the factors that influence this relationship helps you make informed decisions. Several key factors shape whether a task requires a low or high rank.

Distance from pre-training distribution: Tasks requiring the model to behave very differently from its pre-training typically benefit from higher ranks. The model needs to learn substantial new patterns rather than slightly adjust existing ones. When the target behavior diverges significantly from what the model learned during pre-training, the required transformation has higher intrinsic dimensionality, necessitating a larger subspace for the adaptation. Adapting an English language model to generate Python code requires encoding syntactic patterns that the model may not have emphasized during pre-training, which calls for more representational capacity.

Task-specific structure: Some tasks have inherently low-rank structure in the transformations they require. Classification problems that distinguish between a small number of categories often require fewer parameters than generation tasks that must produce diverse, fine-grained outputs. A sentiment classifier essentially needs to learn a projection onto a low-dimensional decision boundary, while a creative writing assistant must capture subtle stylistic variations across an expansive output space. The inherent complexity of the target behavior directly influences the dimensionality required to represent the necessary adaptation.

Dataset size: With limited data (hundreds to low thousands of examples), lower ranks act as regularization, preventing overfitting. The constraint of a small subspace limits the model's ability to memorize training examples. Larger datasets can support higher ranks without memorization, as there is sufficient signal to guide learning in a higher-dimensional space. If you have only 200 training examples, a rank-64 adapter has far more capacity than that data can reliably fill, and much of the extra capacity will be devoted to fitting noise rather than signal.

Model scale: Larger models often require relatively smaller ranks as a fraction of their hidden dimensions. A 70B model has richer, more disentangled internal representations than a 7B model. The richer the internal representations, the less the model needs to restructure its computations for a new task, meaning lower-dimensional adaptations are sufficient. This is why practitioners sometimes find that rank 8 is competitive with rank 32 on large models for tasks where rank 32 is clearly beneficial on smaller models.

Let us visualize how rank affects training dynamics using a simple simulation. This experiment creates a synthetic task with known intrinsic dimensionality and observes how different LoRA ranks perform in capturing the required transformation.

The visualization demonstrates a key insight: once the approximation rank reaches the target's true rank of 8, the reconstruction error drops to essentially zero. Additional rank beyond this point provides no benefit for this particular transformation. In practice, the "true rank" of a task-specific adaptation is unknown, but this principle explains why empirical tuning often reveals a clear knee in performance as rank increases. Finding that knee, the point at which increasing rank stops improving validation performance, is the practical art of rank selection.

Let us walk through a concrete calculation to develop intuition for the parameter savings LoRA provides. Suppose you are fine-tuning a LLaMA-7B model, which has a hidden dimension of $d = 4096$ and 32 transformer layers. Each attention block contains four weight matrices: $W_Q$, $W_K$, $W_V$, and $W_O$, each of shape $4096 \times 4096$. Each FFN block contains three matrices for the SwiGLU architecture: a gate projection, an up projection (each $4096 \times 11008$), and a down projection ($11008 \times 4096$).

Step 1: Count the full fine-tuning parameters.

For a single attention weight matrix of size $4096 \times 4096$, full fine-tuning would optimize $4096 \times 4096 = 16{,}777{,}216$ parameters. With four matrices per layer and 32 layers, attention alone accounts for $4 \times 32 \times 16{,}777{,}216 = 2{,}147{,}483{,}648$ parameters. That is over 2 billion parameters for attention alone.

Step 2: Apply the LoRA parameter formula with rank $r = 16$.

For a single attention weight matrix $W_Q$ of shape $4096 \times 4096$:

$$\text{params}(W_Q) = r \times (d_{in} + d_{out}) = 16 \times (4096 + 4096) = 16 \times 8192 = 131{,}072$$

This is a reduction of $\frac{16{,}777{,}216}{131{,}072} = 128$-fold for that single matrix.

Step 3: Total LoRA parameters across all attention matrices.

With four attention matrices per layer and 32 layers:

$$\text{total attention params} = 4 \times 32 \times 131{,}072 = 16{,}777{,}216$$

That is about 16.8 million trainable parameters for all attention adaptation, compared to over 2 billion for full attention fine-tuning. The ratio is exactly 128, consistent with the single-matrix calculation.

Step 4: Understand memory implications.

At 16-bit (bfloat16) precision, 16.8 million parameters require $16.8 \times 10^6 \times 2$ bytes $\approx 33.6$ MB of memory for the parameters themselves. The optimizer states (Adam maintains first and second moment estimates, each the same size as the parameters) add another $2 \times 33.6 \approx 67.2$ MB. Total adapter-related memory: roughly 100 MB. Compare this to the memory required to store and update optimizer states for 2 billion attention parameters: approximately 12 GB just for optimizer states. This comparison illustrates why LoRA makes fine-tuning accessible on consumer hardware.