LoRA Hyperparameters › LoRA Hyperparameters: Rank, Alpha, and Target Modules

In the previous chapters, we established the mathematical foundation of LoRA and walked through its implementation. You now understand that LoRA introduces trainable low-rank matrices $A$ and $B$ to approximate weight updates without modifying the original model. But a necessary question remains: how do you choose the hyperparameters that govern LoRA's behavior?

The hyperparameters you select fundamentally shape the trade-off between adaptation capacity and efficiency. A rank that is too low might not capture the complexity your task requires; a rank that is too high wastes compute and memory while potentially overfitting. Similarly, choosing which layers to adapt and how to scale their contributions can mean the difference between a well-adapted model and one that fails to generalize. Getting these decisions right matters enormously in practice, and understanding the principles behind each choice makes all the difference between mechanical trial-and-error and informed configuration.

Think of configuring LoRA like choosing the resolution of a sketch. A very low resolution sketch captures only the roughest outlines of the scene, missing important detail. A higher resolution captures nuance and fine structure, but at some point adding more detail provides diminishing returns because the underlying subject only has so much information. The rank parameter is exactly this resolution knob: it sets an upper bound on how much complexity your adaptation can encode. Too low and the adapter is a blurry approximation of what the task demands; too high and you are adding resolution without additional task signal.

The scaling factor alpha operates differently. It does not change the resolution of your sketch, but rather controls how boldly your adaptation is drawn. A high alpha means the adapter's output dominates, strongly overriding the pre-trained model's behavior. A low alpha means the adapter whispers its suggestions, making conservative modifications that preserve most of the original computation. Getting this balance right is as important as choosing the right rank, because even a well-trained adapter at the wrong scaling will produce suboptimal results.

The choice of which layers to adapt determines what aspects of the model's computation you are rewriting. Attention layers control how information flows between positions in the sequence, determining what each token attends to and how it synthesizes that context. Feed-forward layers act more like a database, storing and retrieving factual associations learned during pre-training. Choosing attention, feed-forward, or both commits you to a particular theory of where the task-relevant information and behavior reside in the model. This choice turns out to be highly task-dependent, and understanding the role of each component helps you reason about the best targeting strategy.

This chapter provides practical guidance for configuring LoRA based on both theoretical understanding and empirical findings. We cover each hyperparameter in depth, including the reasoning behind common choices, the interactions between parameters, and concrete recipes for common fine-tuning scenarios. By the end, you will be able to approach a new LoRA task with principled intuitions rather than guesswork.