LoRA Hyperparameters › Target Module Selection

LoRA can be applied to any linear transformation in the model, but which layers you adapt significantly impacts both performance and efficiency. The choice of target modules determines which aspects of the model's computation you are modifying, and different choices lead to qualitatively different adaptations. This is perhaps the most architecturally meaningful hyperparameter decision you will make.

Think of a transformer as two interleaved systems working in concert. The attention subsystem is a dynamic routing mechanism: it decides how much each token should "look at" every other token and combines their representations accordingly. The feed-forward subsystem is a key-value store: it reads the current token representation, looks up relevant patterns memorized during pre-training, and returns corresponding information. When you adapt the attention subsystem, you are changing how information flows. When you adapt the feed-forward subsystem, you are changing what knowledge is retrieved. These are fundamentally different interventions, and the right choice depends on what your task requires.

Recall from Part XV that transformer blocks contain several linear transformations, each playing a distinct role in the model's computation:

• Attention projections: $W_Q$, $W_K$, $W_V$ for queries, keys, and values; $W_O$ for output projection. These matrices control how the model attends to different parts of the input sequence and how it combines information across positions. $W_Q$ and $W_K$ together determine which pairs of tokens have high attention scores. $W_V$ determines what information is extracted from attended positions. $W_O$ determines how the attended information is mixed back into the residual stream.

• Feed-forward networks: Two linear layers (up-projection and down-projection in standard architectures, plus gate projection in gated architectures like SwiGLU). These layers transform each token's representation independently, storing and retrieving factual knowledge. The intermediate representation created by the up-projection activates "memory" neurons that encode specific factual associations, while the down-projection writes the retrieved information back to the residual stream.

Each of these has different roles and different responses to LoRA adaptation. Adapting attention layers primarily affects how the model routes and combines information, while adapting feed-forward layers affects what information is stored and retrieved at each position. Understanding this distinction helps you make targeted choices rather than defaulting to adapting everything.

The original LoRA paper primarily experimented with attention weight matrices and found that adapting $W_Q$ and $W_V$ worked well for many tasks. This focus on attention was partly practical (it kept parameter counts low) and partly theoretical (attention mechanisms are thought to be more task-specific than the general-purpose transformations in feed-forward layers). However, subsequent research has revealed a more fine-grained picture.

The key insight from mechanistic interpretability research is that feed-forward layers often store factual associations more directly than attention layers. Experiments like those in Meng et al. (2022) on "ROME" (Rank-One Model Editing) showed that specific factual associations can be localized to particular FFN layers. If your task requires the model to apply factual knowledge specific to your domain, adapting FFN layers may be more important than adapting attention. Conversely, if your task requires the model to attend to different parts of the input than it would by default (for example, always focusing on the system prompt rather than distributing attention broadly), attention adaptation is the right lever.

The visualization highlights the trade-off between adaptation depth and parameter efficiency. While attention-only strategies remain extremely lightweight, including feed-forward networks (All Linear) increases the parameter count significantly, though it remains a small fraction of the total model size. Notice that FFN-only adaptation uses more parameters than all-attention adaptation, because the feed-forward intermediate dimension (11008 in this example) is much larger than the hidden dimension (4096). This is an important consideration: if parameter budget is your constraint, adapting all attention matrices may give you more targeted coverage than adapting FFN layers alone.

Choosing which modules to target requires balancing task requirements with computational constraints. The following guidelines are organized by strategy and reflect practical experience across many fine-tuning scenarios.

Attention-only (Q, V or all attention): Start here for most tasks. Attention layers determine how information flows between tokens, making them highly influential for task adaptation. When you modify query and value projections, you change what the model looks for in the input and what it extracts from attended positions. The original LoRA paper found Q and V sufficient for many tasks, though including K and O can help for more complex adaptations that require modifying how keys are computed and how attended information is projected back to the residual stream. Adding the K projection allows the model to change what positions are attended to more fundamentally, not just what is looked for. Adding the O projection allows the model to control how attended values are mixed before returning to the residual stream.

Feed-forward layers: These serve as "memory" in transformers, storing factual and procedural knowledge learned during pre-training. Research has shown that specific factual associations are often localized in particular feed-forward layers. Adapting FFN layers becomes important when your task requires new factual knowledge not present in the pre-training data, when you are adapting to a significantly different domain with domain-specific terminology or conventions, or when attention-only adaptation is not achieving desired performance, suggesting the limitation lies in what knowledge is being retrieved rather than how information flows.

All linear layers: The most complete but also most parameter-heavy strategy. Use this when you have abundant training data sufficient to support the increased capacity, when the task requires substantial behavioral changes across multiple dimensions, and when you have validated that simpler strategies underperform and the extra parameters are justified. This strategy is commonly used for ambitious instruction-tuning or domain adaptation projects with large proprietary datasets.

Let us examine how different targeting strategies affect parameter counts in practice using an actual model loaded through the PEFT library.

Focusing on Q and V projections keeps the trainable parameter count under 0.1% of the total model size, representing extreme parameter efficiency. Expanding to all attention matrices roughly doubles this footprint, while including FFN layers (All Linear) increases it nearly five-fold. Even the most complete strategy adapts less than one percent of the model's parameters. This shows LoRA's basic efficiency advantage over full fine-tuning.

An advanced strategy allows different ranks for different layer types. This recognizes that attention and FFN layers may have different intrinsic dimensionalities for adaptation: the patterns needed to modify attention behavior might be simpler (lower rank) than those needed to inject new knowledge into feed-forward layers (higher rank). Rather than treating all weight matrices as equivalent, this approach allocates parameter budget proportionally to each component's likely complexity requirements.