LoRA Hyperparameters: Rank, Alpha & Target Module Selection

In the previous chapters, we established the mathematical foundation of LoRA and walked through its implementation. You now understand that LoRA introduces trainable low-rank matrices $A$ and $B$ to approximate weight updates without modifying the original model. But a necessary question remains: how do you choose the hyperparameters that govern LoRA's behavior?

The hyperparameters you select fundamentally shape the trade-off between adaptation capacity and efficiency. A rank that is too low might not capture the complexity your task requires; a rank that is too high wastes compute and memory while potentially overfitting. Similarly, choosing which layers to adapt and how to scale their contributions can mean the difference between a well-adapted model and one that fails to generalize. Getting these decisions right matters enormously in practice, and understanding the principles behind each choice makes all the difference between mechanical trial-and-error and informed configuration.

Think of configuring LoRA like choosing the resolution of a sketch. A very low resolution sketch captures only the roughest outlines of the scene, missing important detail. A higher resolution captures nuance and fine structure, but at some point adding more detail provides diminishing returns because the underlying subject only has so much information. The rank parameter is exactly this resolution knob: it sets an upper bound on how much complexity your adaptation can encode. Too low and the adapter is a blurry approximation of what the task demands; too high and you are adding resolution without additional task signal.

The scaling factor alpha operates differently. It does not change the resolution of your sketch, but rather controls how boldly your adaptation is drawn. A high alpha means the adapter's output dominates, strongly overriding the pre-trained model's behavior. A low alpha means the adapter whispers its suggestions, making conservative modifications that preserve most of the original computation. Getting this balance right is as important as choosing the right rank, because even a well-trained adapter at the wrong scaling will produce suboptimal results.

The choice of which layers to adapt determines what aspects of the model's computation you are rewriting. Attention layers control how information flows between positions in the sequence, determining what each token attends to and how it synthesizes that context. Feed-forward layers act more like a database, storing and retrieving factual associations learned during pre-training. Choosing attention, feed-forward, or both commits you to a particular theory of where the task-relevant information and behavior reside in the model. This choice turns out to be highly task-dependent, and understanding the role of each component helps you reason about the best targeting strategy.

This chapter provides practical guidance for configuring LoRA based on both theoretical understanding and empirical findings. We cover each hyperparameter in depth, including the reasoning behind common choices, the interactions between parameters, and concrete recipes for common fine-tuning scenarios. By the end, you will be able to approach a new LoRA task with principled intuitions rather than guesswork.

The rank $r$ is LoRA's most important hyperparameter. It determines the dimensionality of the low-rank decomposition and directly controls the number of trainable parameters introduced. Understanding rank requires thinking carefully about what these low-rank matrices represent: they define a compressed subspace within which all adaptation must occur. A higher rank provides a richer, more flexible subspace capable of capturing more fine-grained transformations, while a lower rank constrains the adaptation to simpler patterns that can be expressed with fewer degrees of freedom.

Think of the rank as specifying the number of "directions" in which your adapter can modify the model's behavior. At rank 1, the entire adaptation is captured by a single direction in weight space: the adapter can only make transformations that correspond to scalar multiples of a single outer product. At rank 8, there are eight independent directions available. This allows the adapter to separately encode different aspects of the required transformation. At rank 64, there are 64 directions. This provides the capacity to represent quite complex modifications while still remaining far smaller than the full weight matrix.

The key insight is that for many fine-tuning tasks, the required transformation has low intrinsic dimensionality. Pre-trained models already learn rich, general representations. Task-specific adaptation primarily involves steering these existing capabilities rather than learning entirely new transformations. The model already knows how to process language; fine-tuning on a legal document classification task mostly requires the model to recognize which of its existing linguistic and semantic capabilities to prioritize for that domain. This steering operation turns out to require far fewer dimensions than the original weight matrices would suggest.

Recall from our discussion of LoRA mathematics that for a weight matrix $W \in \mathbb{R}^{d_{out} \times d_{in}}$, LoRA introduces:

where:

• $\Delta W$: the low-rank update matrix that captures all task-specific modifications to the original weights
• $B$: the up-projection matrix ($\in \mathbb{R}^{d_{out} \times r}$), which maps from the compressed rank-$r$ space back to the output dimension
• $A$: the down-projection matrix ($\in \mathbb{R}^{r \times d_{in}}$), which projects the input into the compressed rank-$r$ space

The basic insight here is that the product $BA$ can only produce matrices of rank at most $r$, regardless of how large $d_{in}$ and $d_{out}$ might be. This means we are implicitly assuming that the difference between what the pre-trained model computes and what we need for our specific task can be well-approximated by a low-rank transformation. This assumption is surprisingly valid in practice, as we will see.

Since the adapter consists of matrices $A$ ($r \times d_{in}$) and $B$ ($d_{out} \times r$), the total parameter count is the sum of their elements. To understand this concretely, consider that matrix $A$ has $r$ rows and $d_{in}$ columns, yielding $r \times d_{in}$ individual learnable parameters. Similarly, matrix $B$ has $d_{out}$ rows and $r$ columns, contributing $d_{out} \times r$ parameters. The number of trainable parameters per adapted weight matrix is therefore:

where:

• $\text{params}$: the total count of trainable parameters for the adapter
• $r$: the rank hyperparameter, the single value that controls the capacity of the entire adaptation
• $d_{out}$: the output dimension of the layer being adapted
• $d_{in}$: the input dimension of the layer being adapted

This formula reveals an elegant property: the parameter count scales linearly with rank. Doubling the rank exactly doubles the number of trainable parameters. It also scales linearly with the sum of dimensions rather than their product, which is the key insight that makes LoRA so efficient. For a square weight matrix where $d_{in} = d_{out} = d$, the original weight matrix has $d^2$ parameters while LoRA adds only $2rd$ parameters, a savings factor of $\frac{d}{2r}$. For a model with hidden dimension 4096 and rank 16, this savings factor is 128: the adapter uses roughly 0.78% of the parameters compared to storing the full weight update.

Let us examine how rank affects parameter counts for typical transformer dimensions. This analysis will make concrete the dramatic parameter savings that LoRA provides across different model scales.

The parameter efficiency is striking and reveals why LoRA has become so popular for resource-constrained fine-tuning. For a 4096-dimensional model (like LLaMA-7B's hidden dimension), rank 16 uses about 0.8% of the parameters compared to full fine-tuning of a single weight matrix. This represents a reduction of more than 100x in the parameters that must be stored and optimized, including those distributed across training runs. This efficiency is why LoRA enables fine-tuning on consumer hardware: where full fine-tuning of a 7B parameter model might require multiple high-end GPUs, LoRA adaptation can often be accomplished on a single GPU with 16GB or even less memory.

The x-axis is logarithmic, so each step doubles rank and doubles the LoRA parameter share. For a square $d \times d$ weight matrix, that share is $2r/d$: at the same rank, a larger hidden dimension yields a smaller adapter-to-matrix ratio. The adapter itself is still larger in absolute terms for the larger model because its parameter count grows with $d$.

Research and practitioner experience have converged on general guidelines that provide useful starting points. These recommendations emerge from extensive experimentation across diverse tasks and model architectures. They should be treated as starting points rather than rigid rules, since your specific task and data distribution may justify different choices.

• Rank 4-8: Suitable for simple classification tasks, sentiment analysis, and tasks where the pre-trained model already performs reasonably well. At these ranks, you are making only subtle adjustments to the model's behavior, which is often sufficient when the pre-training distribution closely matches your target domain. For binary sentiment classification of English product reviews, a model pre-trained on large English corpora already understands the language and sentiment concepts; it just needs minor steering.

• Rank 16-32: The sweet spot for most instruction tuning and domain adaptation tasks. These ranks provide enough capacity to learn new behavioral patterns while remaining extremely parameter-efficient. A rank-16 adapter on a 7B model has roughly 4 million trainable parameters, yet this is sufficient to teach the model to follow complex instruction formats, adopt specific output styles, or adapt to a particular domain's vocabulary.

• Rank 64-128: Appropriate for complex tasks requiring significant behavior changes, multilingual adaptation, or when data is abundant. Here, you have capacity comparable to more substantial model modifications. Cross-lingual transfer, for example, may require the adapter to encode a richer set of linguistic correspondences.

• Rank 256+: Rarely necessary; consider whether full fine-tuning might be more appropriate. At these ranks, you are approaching the parameter counts where the low-rank assumption provides diminishing benefits, and the computational savings become smaller relative to the overhead of managing the LoRA infrastructure.

The original LoRA paper found that even rank 4 achieved competitive results on many tasks, suggesting that the update matrices for adapting pre-trained models often have low intrinsic dimensionality. This observation aligns with findings from the intrinsic dimensionality literature, which showed that fine-tuning can be effective in surprisingly low-dimensional subspaces. The implication is that pre-trained models have already learned rich, general representations, and task-specific adaptation primarily involves steering these existing capabilities rather than learning entirely new transformations.

The relationship between task complexity and optimal rank is not strictly linear, and understanding the factors that influence this relationship helps you make informed decisions. Several key factors shape whether a task requires a low or high rank.

Distance from pre-training distribution: Tasks requiring the model to behave very differently from its pre-training typically benefit from higher ranks. The model needs to learn substantial new patterns rather than slightly adjust existing ones. When the target behavior diverges significantly from what the model learned during pre-training, the required transformation has higher intrinsic dimensionality, necessitating a larger subspace for the adaptation. Adapting an English language model to generate Python code requires encoding syntactic patterns that the model may not have emphasized during pre-training, which calls for more representational capacity.

Task-specific structure: Some tasks have inherently low-rank structure in the transformations they require. Classification problems that distinguish between a small number of categories often require fewer parameters than generation tasks that must produce diverse, fine-grained outputs. A sentiment classifier essentially needs to learn a projection onto a low-dimensional decision boundary, while a creative writing assistant must capture subtle stylistic variations across an expansive output space. The inherent complexity of the target behavior directly influences the dimensionality required to represent the necessary adaptation.

Dataset size: With limited data (hundreds to low thousands of examples), lower ranks act as regularization, preventing overfitting. The constraint of a small subspace limits the model's ability to memorize training examples. Larger datasets can support higher ranks without memorization, as there is sufficient signal to guide learning in a higher-dimensional space. If you have only 200 training examples, a rank-64 adapter has far more capacity than that data can reliably fill, and much of the extra capacity will be devoted to fitting noise rather than signal.

Model scale: Larger models often require relatively smaller ranks as a fraction of their hidden dimensions. A 70B model has richer, more disentangled internal representations than a 7B model. The richer the internal representations, the less the model needs to restructure its computations for a new task, meaning lower-dimensional adaptations are sufficient. This is why practitioners sometimes find that rank 8 is competitive with rank 32 on large models for tasks where rank 32 is clearly beneficial on smaller models.

Let us visualize how rank affects training dynamics using a simple simulation. This experiment creates a synthetic task with known intrinsic dimensionality and observes how different LoRA ranks perform in capturing the required transformation.

The visualization demonstrates a key insight: once the approximation rank reaches the target's true rank of 8, the reconstruction error drops to essentially zero. Additional rank beyond this point provides no benefit for this particular transformation. In practice, the "true rank" of a task-specific adaptation is unknown, but this principle explains why empirical tuning often reveals a clear knee in performance as rank increases. Finding that knee, the point at which increasing rank stops improving validation performance, is the practical art of rank selection.

Let us walk through a concrete calculation to develop intuition for the parameter savings LoRA provides. Suppose you are fine-tuning a LLaMA-7B model, which has a hidden dimension of $d = 4096$ and 32 transformer layers. Each attention block contains four weight matrices: $W_Q$, $W_K$, $W_V$, and $W_O$, each of shape $4096 \times 4096$. Each FFN block contains three matrices for the SwiGLU architecture: a gate projection, an up projection (each $4096 \times 11008$), and a down projection ($11008 \times 4096$).

Step 1: Count the full fine-tuning parameters.

For a single attention weight matrix of size $4096 \times 4096$, full fine-tuning would optimize $4096 \times 4096 = 16{,}777{,}216$ parameters. With four matrices per layer and 32 layers, attention alone accounts for $4 \times 32 \times 16{,}777{,}216 = 2{,}147{,}483{,}648$ parameters. That is over 2 billion parameters for attention alone.

Step 2: Apply the LoRA parameter formula with rank $r = 16$.

For a single attention weight matrix $W_Q$ of shape $4096 \times 4096$:

This is a reduction of $\frac{16{,}777{,}216}{131{,}072} = 128$-fold for that single matrix.

Step 3: Total LoRA parameters across all attention matrices.

With four attention matrices per layer and 32 layers:

That is about 16.8 million trainable parameters for all attention adaptation, compared to over 2 billion for full attention fine-tuning. The ratio is exactly 128, consistent with the single-matrix calculation.

Step 4: Understand memory implications.

At 16-bit (bfloat16) precision, 16.8 million parameters require $16.8 \times 10^6 \times 2$ bytes $\approx 33.6$ MB of memory for the parameters themselves. The optimizer states (Adam maintains first and second moment estimates, each the same size as the parameters) add another $2 \times 33.6 \approx 67.2$ MB. Total adapter-related memory: roughly 100 MB. Compare this to the memory required to store and update optimizer states for 2 billion attention parameters: approximately 12 GB just for optimizer states. This comparison illustrates why LoRA makes fine-tuning accessible on consumer hardware.

The scaling factor $\alpha$ (alpha) is LoRA's second key hyperparameter. While rank determines the capacity of the adaptation, alpha controls the magnitude of the adaptation's contribution to the model's output. The adapted layer output $h$ sums the frozen pre-trained projection with the scaled low-rank update. As discussed in the LoRA mathematics chapter, the formula is:

where:

• $h$: the output vector of the adapted layer, combining both original and adapted computations
• $W$: the frozen pre-trained weight matrix that remains unchanged throughout training
• $x$: the input vector to the layer
• $Wx$: the output from the frozen pre-trained weights, representing the model's original behavior
• $BAx$: the update computed by the low-rank adapter, representing the task-specific modification
• $\alpha$: the scaling constant, a hyperparameter you set before training
• $r$: the rank of the LoRA matrices
• $\frac{\alpha}{r}$: the scaling factor that controls adaptation strength, computed from the ratio of alpha to rank
• $B$: the up-projection matrix that expands the low-rank representation
• $A$: the down-projection matrix that compresses the input

The ratio $\frac{\alpha}{r}$ scales the LoRA output before adding it to the original transformation. This ratio is the key quantity that determines how strongly the adaptation influences the model's predictions relative to its pre-trained behavior. When $\frac{\alpha}{r} = 1$, the adapter's raw output is added directly to the frozen weights' output. When $\frac{\alpha}{r} = 2$, the adapter's contribution is doubled. When $\frac{\alpha}{r} = 0.5$, it is halved.

Think of alpha as a volume knob for the adapter. The rank determines the richness of what the adapter can say, and alpha determines how loudly it says it. A high-rank adapter with low alpha makes fine-grained but quiet suggestions to the model's computation. A low-rank adapter with high alpha makes simple but loud contributions. Both settings can be useful depending on the task, and understanding this separation helps you independently tune capacity and strength.

Alpha serves multiple purposes in the LoRA framework, each addressing a different practical concern.

Learning rate decoupling: By scaling the LoRA output, alpha allows you to use the same learning rate regardless of rank. Without this scaling, changing rank would require retuning the learning rate to achieve similar training dynamics. Consider what happens without the $\frac{\alpha}{r}$ factor: a rank-64 adapter would produce outputs with approximately 8 times the magnitude of a rank-8 adapter (due to the summation over more dimensions), completely changing the effective step size of gradient updates. The alpha scaling normalizes these differences, which makes it possible to run experiments at different ranks without adjusting the learning rate each time.

Initialization stability: The $\frac{\alpha}{r}$ factor compensates for the fact that higher ranks produce outputs with higher variance, since more terms are summed. When matrix $A$ maps the input to $r$ dimensions and matrix $B$ maps those back to the output space, the variance of the resulting transformation scales roughly with $r$. The normalization by $r$ in the denominator counteracts this growth, helping maintain consistent output magnitudes across different rank choices. This matters most early in training before the adapter has learned to produce meaningful outputs.

Adaptation strength: Practically, alpha controls how much the LoRA adaptation influences the model's output relative to the frozen pre-trained weights. A larger alpha amplifies the adapter's contribution, letting more aggressive modification of the model's behavior. A smaller alpha produces more conservative updates, keeping the model closer to its pre-trained state. For tasks where the model already has relevant capabilities, a lower alpha preserves those capabilities while letting refinement. For tasks requiring significant behavioral change, a higher alpha enables more substantial modifications.

The ratio $\frac{\alpha}{r}$ determines the effective scaling and is the quantity that directly influences training behavior. Understanding this ratio helps you reason about different configuration choices. Common configurations include:

• $\alpha = r$: The most common choice, resulting in a scaling factor of 1. The LoRA update contributes with the same magnitude as if there were no explicit scaling. This provides a neutral baseline where the raw adapter output is added directly to the pre-trained computation. Many practitioners find this works well as a default across diverse tasks.

• $\alpha = 2r$: Doubles the LoRA contribution, potentially accelerating adaptation but risking instability. With stronger updates, the model can adapt more quickly, but gradients may become large enough to cause training difficulties. This configuration is useful when you observe slow adaptation with the default scaling and want to increase the learning signal without changing the optimizer configuration.

• $\alpha = r/2$: Halves the LoRA contribution. This provides more conservative updates. This is useful when you want the model to retain more of its pre-trained behavior, important for tasks where the original capabilities should remain intact, or when training is unstable with default settings and reducing the adaptation strength can restore smooth convergence.

Selecting alpha in practice involves balancing several considerations. Here are guidelines based on common scenarios.

Default choice ($\alpha = r$): Start here. It provides balanced adaptation and has been validated across many tasks. The scaling factor of 1 means the adapter's influence is neither amplified nor diminished, giving you a clean baseline from which to adjust. Most published LoRA results use this setting as it requires no additional justification.

Higher alpha ($\alpha > r$): Consider when you need faster adaptation with limited training steps, as stronger updates allow more learning per iteration. Also appropriate when the task requires strong deviation from pre-trained behavior, necessitating larger modifications, or when you are using a low learning rate and want to compensate without changing the optimizer configuration. The most common variant is $\alpha = 2r$, which doubles the effective scaling and is widely used in instruction-tuning recipes.

Lower alpha ($\alpha < r$): Consider when fine-tuning on limited data where extra regularization matters. A lower alpha prevents the model from overwriting its useful pre-trained knowledge too aggressively. Also useful when you want to preserve more of the pre-trained model's behavior, which is important for tasks where the original capabilities should remain intact. If training is unstable with default settings, reducing alpha can restore smooth convergence by reducing the magnitude of parameter updates.

A common pattern in practice is to fix $\alpha = 16$ or $\alpha = 32$ regardless of rank. This means changing rank also changes the effective scaling, which can be useful: lower ranks get relatively stronger updates, partially compensating for their reduced capacity. This approach simplifies hyperparameter tuning by reducing the number of independent variables you need to consider. It also has an intuitive interpretation: you are saying "I want a fixed magnitude of adaptation" and letting the rank control the complexity of that adaptation independently.

With fixed alpha, lower ranks receive stronger per-parameter updates (higher effective scaling), while higher ranks receive more diluted updates. This implicit regularization can help: smaller adapters need more "strength" to achieve meaningful adaptation with their limited capacity. Conversely, larger adapters have enough parameters that each individual parameter can contribute less while still achieving substantial overall modification. The fixed-alpha strategy exploits this natural compensation between rank and scaling.

LoRA can be applied to any linear transformation in the model, but which layers you adapt significantly impacts both performance and efficiency. The choice of target modules determines which aspects of the model's computation you are modifying, and different choices lead to qualitatively different adaptations. This is perhaps the most architecturally meaningful hyperparameter decision you will make.

Think of a transformer as two interleaved systems working in concert. The attention subsystem is a dynamic routing mechanism: it decides how much each token should "look at" every other token and combines their representations accordingly. The feed-forward subsystem is a key-value store: it reads the current token representation, looks up relevant patterns memorized during pre-training, and returns corresponding information. When you adapt the attention subsystem, you are changing how information flows. When you adapt the feed-forward subsystem, you are changing what knowledge is retrieved. These are fundamentally different interventions, and the right choice depends on what your task requires.

Recall from Part XV that transformer blocks contain several linear transformations, each playing a distinct role in the model's computation:

• Attention projections: $W_Q$, $W_K$, $W_V$ for queries, keys, and values; $W_O$ for output projection. These matrices control how the model attends to different parts of the input sequence and how it combines information across positions. $W_Q$ and $W_K$ together determine which pairs of tokens have high attention scores. $W_V$ determines what information is extracted from attended positions. $W_O$ determines how the attended information is mixed back into the residual stream.

• Feed-forward networks: Two linear layers (up-projection and down-projection in standard architectures, plus gate projection in gated architectures like SwiGLU). These layers transform each token's representation independently, storing and retrieving factual knowledge. The intermediate representation created by the up-projection activates "memory" neurons that encode specific factual associations, while the down-projection writes the retrieved information back to the residual stream.

Each of these has different roles and different responses to LoRA adaptation. Adapting attention layers primarily affects how the model routes and combines information, while adapting feed-forward layers affects what information is stored and retrieved at each position. Understanding this distinction helps you make targeted choices rather than defaulting to adapting everything.

The original LoRA paper primarily experimented with attention weight matrices and found that adapting $W_Q$ and $W_V$ worked well for many tasks. This focus on attention was partly practical (it kept parameter counts low) and partly theoretical (attention mechanisms are thought to be more task-specific than the general-purpose transformations in feed-forward layers). However, subsequent research has revealed a more fine-grained picture.

The key insight from mechanistic interpretability research is that feed-forward layers often store factual associations more directly than attention layers. Experiments like those in Meng et al. (2022) on "ROME" (Rank-One Model Editing) showed that specific factual associations can be localized to particular FFN layers. If your task requires the model to apply factual knowledge specific to your domain, adapting FFN layers may be more important than adapting attention. Conversely, if your task requires the model to attend to different parts of the input than it would by default (for example, always focusing on the system prompt rather than distributing attention broadly), attention adaptation is the right lever.

The visualization highlights the trade-off between adaptation depth and parameter efficiency. While attention-only strategies remain extremely lightweight, including feed-forward networks (All Linear) increases the parameter count significantly, though it remains a small fraction of the total model size. Notice that FFN-only adaptation uses more parameters than all-attention adaptation, because the feed-forward intermediate dimension (11008 in this example) is much larger than the hidden dimension (4096). This is an important consideration: if parameter budget is your constraint, adapting all attention matrices may give you more targeted coverage than adapting FFN layers alone.

Choosing which modules to target requires balancing task requirements with computational constraints. The following guidelines are organized by strategy and reflect practical experience across many fine-tuning scenarios.

Attention-only (Q, V or all attention): Start here for most tasks. Attention layers determine how information flows between tokens, making them highly influential for task adaptation. When you modify query and value projections, you change what the model looks for in the input and what it extracts from attended positions. The original LoRA paper found Q and V sufficient for many tasks, though including K and O can help for more complex adaptations that require modifying how keys are computed and how attended information is projected back to the residual stream. Adding the K projection allows the model to change what positions are attended to more fundamentally, not just what is looked for. Adding the O projection allows the model to control how attended values are mixed before returning to the residual stream.

Feed-forward layers: These serve as "memory" in transformers, storing factual and procedural knowledge learned during pre-training. Research has shown that specific factual associations are often localized in particular feed-forward layers. Adapting FFN layers becomes important when your task requires new factual knowledge not present in the pre-training data, when you are adapting to a significantly different domain with domain-specific terminology or conventions, or when attention-only adaptation is not achieving desired performance, suggesting the limitation lies in what knowledge is being retrieved rather than how information flows.

All linear layers: The most complete but also most parameter-heavy strategy. Use this when you have abundant training data sufficient to support the increased capacity, when the task requires substantial behavioral changes across multiple dimensions, and when you have validated that simpler strategies underperform and the extra parameters are justified. This strategy is commonly used for ambitious instruction-tuning or domain adaptation projects with large proprietary datasets.

Let us examine how different targeting strategies affect parameter counts in practice using an actual model loaded through the PEFT library.

Focusing on Q and V projections keeps the trainable parameter count under 0.1% of the total model size, representing extreme parameter efficiency. Expanding to all attention matrices roughly doubles this footprint, while including FFN layers (All Linear) increases it nearly five-fold. Even the most complete strategy adapts less than one percent of the model's parameters. This shows LoRA's basic efficiency advantage over full fine-tuning.

An advanced strategy allows different ranks for different layer types. This recognizes that attention and FFN layers may have different intrinsic dimensionalities for adaptation: the patterns needed to modify attention behavior might be simpler (lower rank) than those needed to inject new knowledge into feed-forward layers (higher rank). Rather than treating all weight matrices as equivalent, this approach allocates parameter budget proportionally to each component's likely complexity requirements.

By allocating higher ranks to the FFN layers (which often require more capacity to store new knowledge) and lower ranks to attention layers (where steering behavior is often simpler), this mixed strategy optimizes the parameter budget. The total count lies between the attention-only and all-linear approaches. This provides a middle ground that acknowledges the different roles these components play. This approach is particularly relevant for domain adaptation tasks where you want to both steer the model's attention behavior and update its factual knowledge base.

LoRA dropout applies dropout to the LoRA branch during training. This provides regularization that can improve generalization. Unlike standard dropout which is applied throughout the network, LoRA dropout specifically targets the adapter pathway. This leaves the frozen pre-trained computation unaffected. This targeted application is intentional: the pre-trained model is already well-regularized by the massive amount of data it was trained on. The adapter, however, is trained on a much smaller dataset and is therefore more susceptible to overfitting.

Dropout is applied after the first matrix multiplication in the LoRA path, introducing stochastic noise during the forward pass:

where:

• $h$: the layer output, combining frozen and adapted computations
• $W$: the frozen pre-trained weight matrix
• $x$: the input vector
• $\alpha$: the scaling constant
• $r$: the rank hyperparameter
• $B$: the up-projection matrix
• $A$: the down-projection matrix
• $\text{Dropout}(\cdot)$: the dropout operation that randomly zeros elements with probability $p$
• $Ax$: the intermediate low-rank projection, which is the target of the dropout operation

During training, elements of $Ax$ are randomly zeroed with probability $p$, and the remaining elements are scaled by $\frac{1}{1-p}$ to maintain expected values. This rescaling is the inverted dropout trick: by scaling up surviving units during training, the expected output magnitude remains the same whether dropout is applied or not. At inference time, dropout is disabled and the full LoRA contribution is used, giving consistent and deterministic predictions.

As we discussed in Part X on dropout, this prevents co-adaptation of features and acts as an ensemble of subnetworks. Each training step effectively uses a different random subset of the adapter's capacity, forcing the model to learn reliable representations that do not rely on any single pathway. In the LoRA context, this means the adapter cannot rely on specific neurons always being active. If the adapter's effectiveness depended on a narrow set of highly active intermediate features, dropout would destroy that dependency and force the adapter to distribute its learned patterns more broadly. The result is a more reliable adapter that generalizes better to unseen examples.

Common dropout rates for LoRA range from 0.0 to 0.1, with several factors influencing the choice.

• Dataset size: Smaller datasets benefit from higher dropout (0.05-0.1) to prevent overfitting, as the regularization helps the model generalize from limited examples. Large datasets may need little to no dropout, as the abundance of training signal naturally prevents memorization.
• Rank: Higher ranks have more capacity to overfit, making dropout more valuable. The additional parameters in a high-rank adapter provide more opportunities for the model to memorize training examples. Very low ranks (4 or less) often work fine without dropout, as their limited capacity already provides implicit regularization.
• Task complexity: Simple classification tasks typically need less regularization than generation tasks. Generation requires the model to produce diverse, coherent outputs, which makes it more susceptible to subtle forms of overfitting that dropout can help prevent.
• Training duration: If you are training for more epochs than the dataset size naturally supports, higher dropout helps prevent the adapter from memorizing the training data during the extra passes.

LoRA dropout interacts with rank and alpha in subtle ways that merit careful consideration.

Rank-dropout interaction: With higher ranks, each forward pass uses a larger absolute number of features even at the same dropout rate. A rank-64 adapter with 10% dropout still uses roughly 58 features on average, while a rank-8 adapter uses only about 7. This means dropout is relatively more aggressive for low-rank configurations. A 10% dropout on a rank-4 adapter leaves only about 3.6 features active on average, which may severely limit the adapter's effective capacity. Consider using lower dropout rates with lower ranks, or reserving dropout primarily for higher-rank configurations where the per-pass capacity reduction is less severe.

Alpha-dropout interaction: Dropout does not change the expected value of the output due to the rescaling, but it does change the gradient variance. Each training step sees a different random subset of the network, leading to gradients that fluctuate around their expected values. Higher alpha amplifies both the mean and variance of gradients, so combining high alpha with high dropout can lead to noisy training. If you observe training instability with high alpha, reducing dropout may help restore smooth convergence. Conversely, if you are using low alpha for conservative updates and want some regularization, adding moderate dropout (0.05) is a safer choice than increasing alpha.

Dropout and dataset size: The regularization benefit of dropout is proportional to how prone the adapter is to overfitting, which in turn depends heavily on the ratio of trainable parameters to training examples. If you have 10,000 training examples and a rank-16 adapter with 4 million parameters, each parameter "sees" the training signal only 2-3 times per epoch. In this regime, dropout provides minimal additional regularization because the dataset itself is the primary constraint. With only 500 training examples and the same adapter, each parameter is highly susceptible to memorizing specific examples, and dropout becomes much more valuable.

Let us consolidate these guidelines into practical configurations for common scenarios. These recipes represent starting points based on accumulated practitioner experience, though optimal configurations may vary with your model and data as well as your requirements. Think of these as reasonable defaults that work well across a range of similar situations, not as universal optima.

The recipes reveal a coherent progression: as task complexity increases and the required behavioral change from the pre-trained model grows larger, both rank and the number of target modules increase. The alpha-to-rank ratio stays at either 1.0 or 2.0. This reflects the practical convention of choosing alpha as a simple multiple of rank rather than tuning it independently. Notice that dropout is higher for the low-data scenario (0.1) and absent for code generation (0.0). This reflects the regularization trade-off: with abundant data, dropout provides little benefit and slightly slows training.

Here is how these hyperparameters come together in practice. This shows the configuration structure you would use with the PEFT library.

The configuration targets all attention matrices with rank 16 and alpha 32, creating a scaling factor of 2. This amplified scaling helps the adapter make meaningful contributions despite its small parameter count. The modest dropout of 0.05 provides light regularization without significantly slowing convergence. This setup balances capacity with stability, which makes it a reliable starting point for instruction tuning tasks on modern decoder-only architectures.

When optimal values are not clear from guidelines, systematic search can help. The key is to search efficiently, focusing computational resources on the hyperparameters most likely to impact performance. Because LoRA training is fast, hyperparameter search is much more practical than for full fine-tuning. Where evaluating one full fine-tuning configuration might take 24 hours on a cluster, evaluating a LoRA configuration might take 1-2 hours on a single GPU, which makes it feasible to evaluate dozens of configurations within a reasonable timeline.

Not all hyperparameters deserve equal search effort. Prioritize in this order:

1. Target modules: The biggest impact on both performance and efficiency. Start with attention-only and expand if needed. This decision fundamentally changes what aspects of the model you are adapting, and getting it right matters more than precise rank selection.

2. Rank: Affects capacity directly. Search powers of 2 (4, 8, 16, 32) as these are natural scales for the problem. Try to identify the knee in the performance-vs-rank curve rather than optimizing to a specific value.

3. Alpha: Usually set relative to rank. Try $\alpha = r$, then $\alpha = 2r$ if more adaptation strength is needed. Rarely needs to be tuned beyond these two choices.

4. Dropout: Fine-tune last. Start with 0.05, increase to 0.1 if overfitting, decrease to 0 if underfitting. The benefit is usually modest unless you are training on very limited data.

A balanced search with 36 configurations is manageable for most projects. Each configuration trains quickly due to LoRA's efficiency, making hyperparameter search practical even on limited hardware. Where full fine-tuning might require days to evaluate a single configuration, LoRA often allows evaluating dozens of configurations in the same timeframe.

When performing hyperparameter search, a few practical principles help you extract maximum information from your experiments. First, evaluate configurations on a held-out validation set using your actual task metric (accuracy for classification, ROUGE for summarization, pass@1 for code) rather than training loss. Training loss can be misleading: a high-rank adapter might have lower training loss but higher validation loss due to overfitting.

Second, monitor training curves rather than just final values. A configuration that converges quickly to good performance is preferable to one that requires careful learning rate tuning to achieve similar results. If you observe a loss curve that drops rapidly then plateaus, the rank is likely sufficient and increasing it will not help. If the curve is still descending at the end of training, consider either more training steps or a slightly higher rank.

Third, be alert to signs of underfitting versus overfitting. A large gap between training and validation performance suggests overfitting, pointing toward lower rank, higher dropout, or lower alpha. Very similar training and validation performance that is below your target suggests underfitting, pointing toward higher rank, more target modules, or a higher alpha. This diagnostic framework helps you interpret results systematically rather than adjusting hyperparameters randomly.

Despite LoRA's flexibility, hyperparameter selection involves inherent trade-offs and limitations that you should keep in mind. Understanding these limitations helps you set realistic expectations and avoid common pitfalls.

The relationship between hyperparameters and performance is not always predictable from first principles. Tasks that seem similar may have different optimal configurations due to subtle differences in the required transformations. A classification task on legal documents might need different settings than one on social media text, even if both are binary classification problems. The domain vocabulary, sentence structure, and implicit knowledge requirements all affect what adaptation the model needs. Legal documents require the model to track complex conditional relationships and specialized terminology, while social media text requires tolerance for informal language and implicit cultural references. These different requirements manifest as different optimal rank and target module choices.

Transfer of hyperparameters across model sizes is imperfect. A rank that works well for a 7B parameter model may be suboptimal for a 70B model, as the intrinsic dimensionality of useful adaptations does not scale linearly with model size. Larger models often require relatively smaller ranks (as a fraction of hidden dimensions) to achieve similar results. This may be because larger models have richer internal representations that require only minor steering for new tasks. The implication is that you should not blindly copy hyperparameter configurations from papers that used different model sizes, even if the task and dataset are similar.

The interaction between LoRA hyperparameters and training hyperparameters (learning rate, batch size, schedule) adds another layer of complexity. The optimal learning rate for rank-8 LoRA differs from rank-64, and the relationship is not simply linear. When changing LoRA hyperparameters, you may need to re-tune training parameters for best results. A useful heuristic: larger rank adapters tend to benefit from lower learning rates, because with more parameters available the optimizer can make fine-grained updates rather than large directional steps. Consider this interaction when interpreting experimental results, especially when comparing configurations from different sources.

An underappreciated limitation is that LoRA hyperparameters that work well for one base model may not transfer to a different architecture even with the same nominal size. Models with different pre-training objectives, architectures (GQA vs. MHA, SwiGLU vs. ReLU FFNs), or training data distributions may respond differently to the same LoRA configuration. When switching from one model family to another, it is worth re-running a basic hyperparameter sweep rather than assuming portability.

Finally, these guidelines assume standard fine-tuning scenarios. Emerging techniques like QLoRA (which we will cover next) introduce additional hyperparameters and interactions. Methods like AdaLoRA dynamically adjust rank during training, partially automating the selection process but introducing their own tuning considerations. As the field continues to evolve, the principles established here, understanding rank as capacity, alpha as scaling, target modules as scope, and dropout as regularization, provide a stable conceptual framework even as specific recommended values may shift with new empirical findings.

The core parameters that govern LoRA configuration can be summarized concisely:

• Rank ($r$): Determines the capacity of the update matrices. Start with 8-16 for simple tasks, increasing to 32-64 for complex adaptations. Look for the knee in the performance-vs-rank curve.

• Alpha ($\alpha$): Controls the strength of the adaptation relative to the pre-trained weights. Often set equal to rank (1x scaling) or double the rank (2x scaling). Fixed alpha values of 16 or 32 across rank choices are also common.

• Target Modules: Determines which model layers are adapted. Attention projections (Q, V) are the standard starting point; FFN layers can be added for increased capacity or domain knowledge injection.

• Dropout: Provides regularization for the adapter. Typical values are 0.05-0.1, with higher values used for smaller datasets. Set to 0 for large, abundant datasets.

The next chapter introduces QLoRA, which combines LoRA with quantization to enable fine-tuning of large models on consumer hardware by storing the frozen model in reduced precision. The hyperparameter principles established here carry over directly to QLoRA, making this chapter's guidelines a foundation for the quantization-aware fine-tuning methods that follow.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about LoRA hyperparameters and their configuration.