LoRA Concept › LoRA Flexibility
Language AI HandbookAll chapters LoRA Concept › LoRA FlexibilityOne section · Sections in this chapter8 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Beyond raw efficiency, LoRA provides flexibility in how and where adaptation is applied. This flexibility is what makes LoRA suitable for such a wide range of use cases, from simple domain adaptation to complex multi-task systems. Understanding these dimensions of flexibility helps you configure LoRA intelligently for your specific situation.
Choosing Which Weights to Adapt
Not all weight matrices contribute equally to task adaptation, and understanding which ones to target is an important part of using LoRA effectively. LoRA can be selectively applied to different components of the model:
- Attention projections: , , , , the query, key, value, and output projection matrices
- Feed-forward networks: The up projection, down projection, and gate projection matrices in each FFN block
- Embedding layers: Input and output token embedding matrices
The original LoRA paper found that adapting the query and value projections ( and ) provided most of the benefit, with diminishing returns from adding more matrices. The intuition is that these projections determine what the model "looks at" (queries) and what information it extracts (values), making them the most task-sensitive components. Subsequent research has produced more fine-grained recommendations: for instruction-following tasks, adapting all four attention projections tends to work best; for domain adaptation, focusing on value and output projections may suffice.
The ability to choose which matrices to adapt also provides a way to balance efficiency against expressiveness. Adapting only and requires the fewest parameters. Adapting all attention matrices provides more expressive power at moderate cost. Adapting both attention and feed-forward matrices provides maximum expressiveness at the cost of more parameters and slightly longer training times.
import torch
import torch.nn as nn
class LoRALinear(nn.Module):
"""A linear layer with LoRA adaptation."""
def __init__(self, in_features, out_features, rank=8, alpha=1.0):
super().__init__()
# Frozen pre-trained weights
self.weight = nn.Parameter(
torch.randn(out_features, in_features), requires_grad=False
)
# Trainable LoRA parameters
self.lora_A = nn.Parameter(torch.randn(rank, in_features) * 0.01)
self.lora_B = nn.Parameter(torch.zeros(out_features, rank))
self.scaling = alpha / rank
def forward(self, x):
# Original computation plus low-rank update
base_output = nn.functional.linear(x, self.weight)
lora_output = (x @ self.lora_A.T @ self.lora_B.T) * self.scaling
return base_output + lora_output
# Example usage
layer = LoRALinear(in_features=4096, out_features=4096, rank=8)
# Calculate parameter counts
trainable = sum(p.numel() for p in layer.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in layer.parameters() if not p.requires_grad)
trainable_fraction = trainable / (trainable + frozen) * 100Trainable parameters: 65,536 Frozen parameters: 16,777,216 Trainable fraction: 0.39%
This result highlights the extreme efficiency of LoRA. Making less than 1% of the parameters trainable significantly reduces the computational burden of gradient updates while maintaining the ability to adapt the model. The frozen weight matrix accounts for the large majority of parameters and is responsible for computing the rich pre-trained representations, while the tiny adapter matrices direct and refine those representations for the specific task.
Multiple Adapters for Multiple Tasks
Since LoRA adapters are small and independent, a single base model can support many task-specific adapters simultaneously. This property opens up deployment architectures that simply do not exist with full fine-tuning.
Consider a company that needs to serve a customer service assistant, a code review helper, a document summarizer, and a translation service, all built on the same base language model. With full fine-tuning, they would need four separate model copies totaling 56GB of storage, four separate inference processes, and four separate serving stacks. With LoRA, they need one base model and four 8MB adapter files. At request time, the serving system loads the appropriate adapter, performs the forward pass with , and returns the response. Switching between adapters requires only swapping two small matrices, which takes milliseconds.
This multi-adapter paradigm enables scenarios including:
- Multi-tenant serving: Different customers or user groups receive personalized model behavior from the same base
- A/B testing: Different fine-tuning approaches can be compared by swapping adapters without changing the serving infrastructure
- Ensemble methods: Multiple adapters trained on different data distributions can be combined to handle diverse inputs
- Progressive specialization: A general-purpose adapter can be used as a starting point and further refined for specialized sub-tasks
The adapters can be swapped at runtime without reloading the base model, letting flexible deployment architectures that scale to hundreds of tasks with a fixed base model footprint.
Rank as an Expressiveness Knob
The rank provides a continuous tradeoff between efficiency and expressiveness. Higher ranks allow capturing more complex adaptations but require more parameters. The relationship is straightforward: doubling the rank roughly doubles the number of adapter parameters, while potentially letting the model to capture more fine-grained task-specific adjustments.
The optimal rank depends on several factors:
-
Task complexity: Simple classification tasks often work well with rank 4 or 8. Complex generation tasks like coding or mathematical reasoning may benefit from rank 16-64. Tasks that require memorizing specific factual patterns may need higher ranks still.
-
Model size: Larger models generally have more redundancy in their weights and may be better compressed. A 70B model might achieve good results with lower ranks than a 7B model needs for the same task.
-
Dataset size: With limited fine-tuning data, lower ranks act as stronger regularization, reducing the risk of overfitting. With large, diverse datasets, higher ranks can be used safely.
-
Compute budget: If training compute is the bottleneck, lower ranks finish faster. If the bottleneck is achieving maximum quality within a compute budget, higher ranks may be worth it.
The visualization shows that LoRA parameters scale linearly with rank. This provides a predictable tradeoff between expressiveness and efficiency. At rank 64, the adapter share ranges from about 1.6% for the 8192-wide example to 16.7% for the 768-wide example. The same rank therefore consumes a smaller fraction of the adapted matrices as hidden size grows. This linear scaling makes it easy to adjust the rank based on task requirements and available compute resources: doubling the rank doubles the parameter cost, so the tradeoff is explicit.