Adapter Layers › Worked Example

Let's trace through an adapter computation manually to build deep intuition for how the bottleneck transformation works. Following the computation step by step reveals how each component contributes to the final output, verifies our understanding of the mathematics, and clarifies the interaction between compression, nonlinearity, and the residual connection. Working through the numbers by hand is particularly valuable for understanding what "near-zero initialization" means concretely and why the residual produces small, targeted modifications rather than wholesale replacement of the representation.

Consider a simplified scenario with hidden dimension $d = 4$ and bottleneck dimension $r = 2$. While real adapters operate in much higher dimensions, this small example allows us to see every number in the computation and check each step. We begin with an input hidden state representing a token in our transformer:

$$\mathbf{h} = \begin{bmatrix} 0.5 \\ -0.3 \\ 0.8 \\ -0.1 \end{bmatrix}$$

This vector might represent, for example, the token "review" in the sentence "Write a product review." The four dimensions capture different aspects of the token's meaning and context as computed by the layers above. Our adapter has the following weights, which we treat as fixed for this example but would in practice be learned through gradient descent:

$$\mathbf{W}_{down} = \begin{bmatrix} 0.4 & 0.2 & -0.1 & 0.5 \\ -0.2 & 0.5 & 0.3 & 0.1 \end{bmatrix}, \quad \mathbf{b}_{down} = \begin{bmatrix} 0.1 \\ -0.1 \end{bmatrix}$$ $$\mathbf{W}_{up} = \begin{bmatrix} 0.3 & -0.2 \\ 0.1 & 0.4 \\ -0.2 & 0.2 \\ 0.5 & -0.1 \end{bmatrix}, \quad \mathbf{b}_{up} = \begin{bmatrix} 0 \\ 0 \\ 0 \\ 0 \end{bmatrix}$$

The zero bias in $\mathbf{b}_{up}$ reflects the near-zero initialization strategy. At initialization, both weight matrices would be set to small random values close to zero. This keeps the adapter contributes nearly nothing to start.

Step 1: Down-projection

The first step compresses the 4-dimensional input to a 2-dimensional bottleneck representation. This linear transformation identifies which combinations of input features are most relevant for the task. We compute the matrix-vector product by taking the dot product of each row of $\mathbf{W}_{down}$ with the input vector $\mathbf{h}$:

$$\mathbf{W}_{down} \cdot \mathbf{h} = \begin{bmatrix} 0.4(0.5) + 0.2(-0.3) + (-0.1)(0.8) + 0.5(-0.1) \\ (-0.2)(0.5) + 0.5(-0.3) + 0.3(0.8) + 0.1(-0.1) \end{bmatrix}$$

Computing each element explicitly:

$$\begin{aligned} z_1 &= 0.20 - 0.06 - 0.08 - 0.05 = 0.01 \\ z_2 &= -0.10 - 0.15 + 0.24 - 0.01 = -0.02 \end{aligned}$$

Adding the bias shifts the compressed representation:

$$\mathbf{z} = \begin{bmatrix} 0.01 + 0.1 \\ -0.02 - 0.1 \end{bmatrix} = \begin{bmatrix} 0.11 \\ -0.12 \end{bmatrix}$$

The down-projection has compressed our 4-dimensional input into a 2-dimensional representation. The values are small, which is typical for a nearly-initialized adapter: the weight matrix is not yet "tuned" to emphasize large variations in the input, so the compressed representation is close to the bias.

Step 2: Nonlinearity (ReLU)

The activation function introduces nonlinearity, letting the adapter to model complex patterns that no linear transformation could capture. ReLU zeroes out negative values while passing positive values unchanged:

$$f(\mathbf{z}) = \text{ReLU}(\mathbf{z}) = \begin{bmatrix} \max(0, 0.11) \\ \max(0, -0.12) \end{bmatrix} = \begin{bmatrix} 0.11 \\ 0 \end{bmatrix}$$

Notice that the second dimension is zeroed by ReLU, because $-0.12 < 0$. Effectively, only one of the two bottleneck dimensions contributes to the output at this point in our example. This sparsity is characteristic of ReLU networks: negative activations are simply shut off, creating a selective routing of information. Depending on the input, different bottleneck dimensions will be active, letting the adapter to implement different transformations for different types of inputs. This context-dependent routing is a key reason why the nonlinearity is needed: without it, the adapter would apply exactly the same linear transformation to every input, regardless of content.

Step 3: Up-projection

The final transformation expands the activated bottleneck representation back to the original hidden dimension. Since the second dimension of $f(\mathbf{z})$ is zero, only the first column of $\mathbf{W}_{up}$ contributes to the output, making the computation particularly clean:

$$\text{Adapter}(\mathbf{h}) = \mathbf{W}_{up} \cdot f(\mathbf{z}) + \mathbf{b}_{up}$$ $$= \begin{bmatrix} 0.3(0.11) + (-0.2)(0) \\ 0.1(0.11) + 0.4(0) \\ (-0.2)(0.11) + 0.2(0) \\ 0.5(0.11) + (-0.1)(0) \end{bmatrix} + \begin{bmatrix} 0 \\ 0 \\ 0 \\ 0 \end{bmatrix} = \begin{bmatrix} 0.033 \\ 0.011 \\ -0.022 \\ 0.055 \end{bmatrix}$$

The up-projection produces a 4-dimensional output. Each element is the scalar multiple of 0.11 times the corresponding entry in the first column of $\mathbf{W}_{up}$. The result is a small modification vector: positive adjustments to dimensions 1, 2, and 4; a small negative adjustment to dimension 3. This pattern reflects the task-specific correction the adapter has learned to apply.

Step 4: Residual connection

Finally, the adapter output is added to the original input through the residual connection, combining the pretrained representation with the task-specific modification:

$$\mathbf{h}' = \mathbf{h} + \text{Adapter}(\mathbf{h}) = \begin{bmatrix} 0.5 \\ -0.3 \\ 0.8 \\ -0.1 \end{bmatrix} + \begin{bmatrix} 0.033 \\ 0.011 \\ -0.022 \\ 0.055 \end{bmatrix} = \begin{bmatrix} 0.533 \\ -0.289 \\ 0.778 \\ -0.045 \end{bmatrix}$$

The adapter has made small, targeted adjustments to each dimension of the hidden state. Notice how the original structure is preserved: the values remain in roughly the same range as before, with only modest shifts in each dimension. The third dimension decreased slightly (0.800 to 0.778), while the fourth changed most dramatically relative to its original value (from $-0.100$ to $-0.045$, almost halving the magnitude of the negative value). The residual connection ensures these changes are additive modifications rather than complete replacements, which is the basic mechanism that allows adapters to retain pretrained knowledge while learning new behaviors.

This example uses a nearly-converged adapter (weights are small but non-trivial). For a freshly initialized adapter with weights close to zero, the modification vector would be much closer to $\mathbf{0}$, and $\mathbf{h}' \approx \mathbf{h}$. The adapter gradually grows its influence through training as gradients accumulate.