LoRA Hyperparameters › Limitations and Practical Considerations

Despite LoRA's flexibility, hyperparameter selection involves inherent trade-offs and limitations that you should keep in mind. Understanding these limitations helps you set realistic expectations and avoid common pitfalls.

The relationship between hyperparameters and performance is not always predictable from first principles. Tasks that seem similar may have different optimal configurations due to subtle differences in the required transformations. A classification task on legal documents might need different settings than one on social media text, even if both are binary classification problems. The domain vocabulary, sentence structure, and implicit knowledge requirements all affect what adaptation the model needs. Legal documents require the model to track complex conditional relationships and specialized terminology, while social media text requires tolerance for informal language and implicit cultural references. These different requirements manifest as different optimal rank and target module choices.

Transfer of hyperparameters across model sizes is imperfect. A rank that works well for a 7B parameter model may be suboptimal for a 70B model, as the intrinsic dimensionality of useful adaptations does not scale linearly with model size. Larger models often require relatively smaller ranks (as a fraction of hidden dimensions) to achieve similar results. This may be because larger models have richer internal representations that require only minor steering for new tasks. The implication is that you should not blindly copy hyperparameter configurations from papers that used different model sizes, even if the task and dataset are similar.

The interaction between LoRA hyperparameters and training hyperparameters (learning rate, batch size, schedule) adds another layer of complexity. The optimal learning rate for rank-8 LoRA differs from rank-64, and the relationship is not simply linear. When changing LoRA hyperparameters, you may need to re-tune training parameters for best results. A useful heuristic: larger rank adapters tend to benefit from lower learning rates, because with more parameters available the optimizer can make fine-grained updates rather than large directional steps. Consider this interaction when interpreting experimental results, especially when comparing configurations from different sources.

An underappreciated limitation is that LoRA hyperparameters that work well for one base model may not transfer to a different architecture even with the same nominal size. Models with different pre-training objectives, architectures (GQA vs. MHA, SwiGLU vs. ReLU FFNs), or training data distributions may respond differently to the same LoRA configuration. When switching from one model family to another, it is worth re-running a basic hyperparameter sweep rather than assuming portability.

Finally, these guidelines assume standard fine-tuning scenarios. Emerging techniques like QLoRA (which we will cover next) introduce additional hyperparameters and interactions. Methods like AdaLoRA dynamically adjust rank during training, partially automating the selection process but introducing their own tuning considerations. As the field continues to evolve, the principles established here, understanding rank as capacity, alpha as scaling, target modules as scope, and dropout as regularization, provide a stable conceptual framework even as specific recommended values may shift with new empirical findings.