Adapter Layers › Adapter Dimensionality

The bottleneck dimension $r$ is the primary hyperparameter controlling the trade-off between adapter capacity and parameter efficiency. Choosing the right value requires understanding how information flows through the bottleneck and how the degree of compression affects the adapter's ability to learn useful transformations. Too small a bottleneck and the adapter cannot encode enough task-specific information; too large and we lose the regularization benefits and approach the cost of full fine-tuning.

Think of the bottleneck dimension as the bandwidth of a communication channel between the input and output of the adapter. With a narrow channel (small $r$), the adapter must ruthlessly prioritize: only the most consistently useful task-relevant patterns can make it through, and idiosyncratic or noisy features are filtered out. With a wider channel (large $r$), the adapter has more freedom to encode complex transformations, but also more opportunity to overfit to training-set peculiarities. The right channel width depends on how much information needs to flow through it for the task at hand.

When we compress a hidden representation from dimension $d$ to dimension $r$, we are forcing the adapter to identify and preserve the most task-relevant information. This compression acts as a form of regularization, preventing the adapter from memorizing noise in the training data. The principle at work connects to information theory: by limiting the capacity of the channel through which task information must flow, we force the adapter to be selective about what it encodes.

Consider a hidden representation $\mathbf{h} \in \mathbb{R}^{768}$ compressed to $r = 64$ dimensions. The adapter must encode whatever task-specific transformation it needs to make using only 64 degrees of freedom. This constraint encourages the adapter to learn generalizable patterns rather than fitting to idiosyncrasies in the training set. If a particular feature of the training data is not consistently useful for the task, the limited bottleneck capacity will prevent the adapter from dedicating parameters to capturing it.

The compression also has implications for what kinds of transformations adapters can represent. With a 12:1 compression ratio, the adapter cannot simply learn an arbitrary linear transformation of the input. Instead, it must identify a low-dimensional subspace that captures the needed variation needed for task adaptation. This constraint is often beneficial, as it biases the adapter toward learning transformations that align with the structure already present in the pretrained representations. If the pretrained model has organized its representations so that semantically similar inputs are nearby in representation space, the adapter's bottleneck will naturally pick up on this structure and use it, rather than learning a transformation that ignores the pretrained model's organization.

The compression ratio also interacts with the dimensionality of the task itself. A simple binary classification task, such as positive or negative sentiment, might require only a handful of dimensions to adapt; the model needs to learn which features to attend to and how to weight them for the binary decision. A complex structured prediction task, like named entity recognition across dozens of entity types, requires more fine-grained adjustments and may benefit from a larger bottleneck. The bottleneck dimension should roughly match the intrinsic complexity of the adaptation, not the dimensionality of the representation space.

Research and practical experience suggest the following guidelines for choosing adapter dimensions. The appropriate choice depends on task complexity, data availability, and computational constraints:

• Very small ($r = 8$ to $16$): Suitable for tasks closely related to pretraining, such as adapting a language model for a slightly different text domain, or when labeled data is extremely limited. Provides strong regularization but limited expressiveness. These minimal adapters work best when the task primarily requires recombining existing features rather than learning fundamentally new representations. If you have fewer than 1,000 training examples, consider starting here.
• Small ($r = 32$ to $64$): A good default for most NLP tasks. Balances efficiency with sufficient capacity for meaningful adaptation. This range provides enough flexibility to learn task-specific transformations while maintaining strong regularization. The original Houlsby et al. paper used $r = 64$ for most experiments and found strong performance across diverse benchmarks.
• Medium ($r = 128$ to $256$): Appropriate for complex tasks or when full fine-tuning achieves significantly better results and you need to close the gap. Approaches the capacity of larger modifications. Consider this range when standard adapter sizes underperform or when the task requires substantial deviation from pretrained behavior, such as adapting to a highly specialized domain with very different vocabulary and linguistic patterns.
• Large ($r \geq 512$): Rarely necessary; at this point, the parameter savings over full fine-tuning diminish substantially. If this much capacity is needed, alternative methods or full fine-tuning may be more appropriate. An adapter with $r = 512$ for BERT-base adds roughly 800K parameters per adapter, which over 12 layers with dual placement exceeds 19 million parameters, about 17% of the base model.

When choosing a bottleneck dimension, we recommend starting with $r = 64$ and evaluating on a validation set. If validation performance plateaus well below full fine-tuning, try increasing to $r = 128$ or $r = 256$. If you are concerned about overfitting on a small dataset, try reducing to $r = 32$ or $r = 16$. The sensitivity of performance to this choice varies by task, so a quick sweep over a few values is usually worthwhile.

You should consider how to scale adapter dimensions as model size increases. Larger models have larger hidden dimensions, so maintaining the same absolute bottleneck size results in increasingly severe compression. A bottleneck of 64 represents a 12:1 compression ratio for BERT-base (768-dimensional hidden states) but a 64:1 ratio for LLaMA-7B (4096-dimensional hidden states). That 64:1 ratio may be too aggressive, leaving insufficient capacity for meaningful adaptation.

A useful heuristic is to maintain a consistent compression ratio $d/r$. For example:

However, empirical tuning often reveals that smaller models need proportionally larger adapters than bigger models. This may relate to the observation that larger models have more redundant capacity and can express task-specific information more compactly. A 7-billion parameter model has learned richer internal representations that can more easily be combined and adjusted through a small adapter, while a smaller model may require more adapter capacity to achieve the same effect. When in doubt, treating the compression ratio as a starting point and validating empirically will serve you better than following any fixed rule.

Let's compare the parameter overhead of adapters at different bottleneck dimensions for a BERT-base model with 12 layers and hidden dimension 768:

Even at $r = 256$, adapters add less than 10% of the base model's parameters. This efficiency enables practical storage of dozens of task-specific adapters alongside a single frozen base model. For organizations deploying models across many tasks, this reduces storage and memory use as well as model-management overhead compared to maintaining separate fully fine-tuned models for each task. A fleet of 50 task adapters at $r = 64$ would together consume only about 120 million parameters, roughly equal to one copy of BERT-base, while supporting 50 completely different tasks.