Bahdanau Attention › The Attention Mechanism: From Intuition to Formulas

With the alignment problem clearly defined, we can now develop the mathematical machinery that Bahdanau attention uses to solve it. Our journey will take us through three interconnected concepts: computing relevance scores, converting those scores to probabilities, and aggregating information based on those probabilities. Each piece builds naturally on the previous one, culminating in a mechanism that allows the decoder to "look back" at exactly the right parts of the input at each generation step.

Attention addresses a simple question: when the decoder is about to generate output token $i$, how relevant is each encoder position $j$ to that decision? If we could answer this question with a single number for each encoder position, we'd have a way to rank and weight the encoder states.

But what should this "relevance score" capture? Consider translating "The cat sat on the mat" to French. When generating "chat" (cat), the decoder needs to know:

• What the decoder is currently looking for (encoded in its hidden state $s_{i-1}$)
• What information each encoder position offers (encoded in hidden states $h_1, h_2, \ldots, h_{T_x}$)

The relevance of position $j$ depends on both of these: what $h_j$ contains and whether that content matches what the decoder currently needs. This suggests we need a function that takes both the decoder state and an encoder state as inputs and produces a scalar score.

Bahdanau and colleagues designed a solution called the additive (or concatenative) alignment model. Rather than directly comparing the decoder and encoder states, which might have different dimensions and represent different types of information, they project both into a shared "alignment space" where comparison becomes meaningful.

The alignment score between decoder position $i$ and encoder position $j$ is computed as:

$$e_{ij} = v_a^\top \tanh(W_a s_{i-1} + U_a h_j)$$

where:

• $e_{ij}$: the alignment score, a scalar indicating how relevant encoder position $j$ is for generating output $i$
• $s_{i-1} \in \mathbb{R}^{d_s}$: the decoder hidden state at the previous timestep, encoding what the decoder "knows" and "needs"
• $h_j \in \mathbb{R}^{d_h}$: the encoder hidden state at position $j$, encoding information about input token $j$ and its context
• $W_a \in \mathbb{R}^{d_a \times d_s}$: a learnable weight matrix that projects the decoder state into the alignment space
• $U_a \in \mathbb{R}^{d_a \times d_h}$: a learnable weight matrix that projects encoder states into the same alignment space
• $v_a \in \mathbb{R}^{d_a}$: a learnable vector that reduces the combined representation to a scalar
• $d_a$: the dimension of the alignment space (a hyperparameter, typically matching the hidden dimensions)

This formula might look complex at first glance, but it follows a clear logic. Let's trace through it step by step to understand why each component is necessary:

Step 1: Project into alignment space. The decoder state $s_{i-1}$ and encoder state $h_j$ live in potentially different vector spaces with different dimensions. Before comparing them, we need to transform them into a common representation. The matrices $W_a$ and $U_a$ learn these transformations during training:

• $W_a s_{i-1}$ produces a $d_a$-dimensional vector representing "what the decoder is looking for"
• $U_a h_j$ produces a $d_a$-dimensional vector representing "what encoder position $j$ offers"

Step 2: Combine additively. With both vectors in the same space, we add them element-wise: $W_a s_{i-1} + U_a h_j$. This additive combination allows the model to detect when the decoder's query and the encoder's content are compatible. If certain dimensions align well (both positive or both negative), they reinforce each other; if they conflict, they cancel out.

Step 3: Apply nonlinearity. The $\tanh$ function serves two purposes. First, it bounds the values to $[-1, 1]$, preventing any single dimension from dominating. Second, it introduces nonlinearity, enabling the model to learn complex, non-linear relationships between decoder queries and encoder contents. Without this nonlinearity, the entire alignment model would collapse to a simple linear function.

Why "additive" rather than some other approach? Additive combination with separate projection matrices gives the model maximum flexibility. The decoder and encoder might have different hidden dimensions, and even if they're the same size, they encode different types of information. By learning separate projections $W_a$ and $U_a$, the model can transform each representation into whatever form makes comparison most effective. Bahdanau and colleagues found this approach worked well in practice, and it became the foundation for subsequent attention mechanisms.

We now have a way to compute alignment scores $e_{ij}$ for each encoder position. But these raw scores can be any real number: positive, negative, or zero. To use them for weighting encoder states, we need to transform them into something more interpretable: a probability distribution over encoder positions.

This is where the softmax function enters the picture. Softmax takes a vector of arbitrary real numbers and converts it into a valid probability distribution where all values are positive and sum to 1:

$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}$$

where:

• $\alpha_{ij}$: the attention weight for encoder position $j$ when generating output $i$
• $e_{ij}$: the raw alignment score from our scoring function
• $\exp(\cdot)$: the exponential function ($e^x$)
• $T_x$: the length of the source sequence
• $\sum_{k=1}^{T_x} \exp(e_{ik})$: the normalizing constant that ensures weights sum to 1

The exponential function is the key to softmax's behavior. It has two essential properties that make it ideal for attention:

1. Positivity: $\exp(x) > 0$ for all real $x$, guaranteeing non-negative weights regardless of the input scores
2. Monotonicity with amplification: larger scores produce exponentially larger values, which amplifies differences between scores

This second property is particularly important. Consider two scores: $e_{i1} = 1.2$ and $e_{i2} = 3.5$. The difference looks modest, but after exponentiation, $\exp(3.5) \approx 33$ while $\exp(1.2) \approx 3.3$, a 10x ratio. Softmax creates a "winner-take-most" dynamic where the highest-scoring positions dominate, while still allowing lower-scoring positions to contribute.

The attention weight $\alpha_{ij}$ has a clear interpretation: it represents the probability that encoder position $j$ contains the information most relevant for generating output token $i$. When we compute attention, we're asking: "Given what the decoder currently needs, where in the input should it look?"

The entropy values quantify this difference: higher entropy indicates more spread-out attention, while lower entropy indicates more concentrated attention. A well-trained model learns to produce focused attention when one input position is clearly relevant (e.g., translating a content word) and spread attention when multiple positions contribute (e.g., translating a phrase that spans multiple source words).

We've now computed attention weights $\alpha_{ij}$ that tell us how much the decoder should attend to each encoder position. The final step is to use these weights to create a single vector that summarizes the relevant information from the entire input sequence. This is the context vector.

The context vector for decoder position $i$ is simply a weighted sum of the encoder hidden states:

$$c_i = \sum_{j=1}^{T_x} \alpha_{ij} h_j$$

where:

• $c_i \in \mathbb{R}^{d_h}$: the context vector, a $d_h$-dimensional summary of the relevant input information
• $\alpha_{ij}$: the attention weight for encoder position $j$ (how much to attend to that position)
• $h_j \in \mathbb{R}^{d_h}$: the encoder hidden state at position $j$
• $T_x$: the length of the source sequence

This formula is simple, but its implications are significant. Because the attention weights sum to 1, the context vector is a convex combination of the encoder states. It lies somewhere "between" them in the vector space, closer to the states with higher weights.

Consider what happens in different scenarios:

• Focused attention: If $\alpha_{i2} = 0.95$ and all other weights are near zero, then $c_i \approx h_2$. The context vector copies the encoder state at position 2.
• Distributed attention: If weights are spread across multiple positions, the context vector blends information from all of them. This is useful when the decoder needs information from multiple parts of the input.
• Uniform attention: If all weights are equal ($\alpha_{ij} = 1/T_x$), the context vector is the simple average of all encoder states, similar to a mean-pooling operation.

We've now developed all three components of Bahdanau attention:

1. Scoring: $e_{ij} = v_a^\top \tanh(W_a s_{i-1} + U_a h_j)$ measures relevance
2. Normalization: $\alpha_{ij} = \text{softmax}(e_{ij})$ converts scores to probabilities
3. Aggregation: $c_i = \sum_j \alpha_{ij} h_j$ creates a weighted summary

But how does this attention mechanism integrate with the decoder RNN? The context vector $c_i$ contains a summary of the relevant input information, but the decoder also needs to know what it has generated so far and maintain its own internal state.

In Bahdanau's formulation, these three sources of information are combined by feeding them into the decoder RNN:

$$s_i = f(s_{i-1}, y_{i-1}, c_i)$$

where:

• $s_i \in \mathbb{R}^{d_s}$: the new decoder hidden state after processing step $i$
• $s_{i-1} \in \mathbb{R}^{d_s}$: the previous decoder hidden state (the decoder's "memory")
• $y_{i-1} \in \mathbb{R}^{d_e}$: the embedding of the previous output token
• $c_i \in \mathbb{R}^{d_h}$: the context vector from attention
• $f$: a recurrent cell function (typically GRU or LSTM)

In practice, $y_{i-1}$ and $c_i$ are concatenated into a single vector of dimension $d_e + d_h$ before being fed to the RNN. This combines information about the previous prediction with the attention-weighted summary of the input.

The complete decoding process at step $i$ follows this sequence:

1. Compute alignment scores: For each encoder position $j$, calculate $e_{ij} = v_a^\top \tanh(W_a s_{i-1} + U_a h_j)$
2. Normalize to attention weights: Apply softmax to get $\alpha_{ij}$
3. Compute context vector: Calculate $c_i = \sum_j \alpha_{ij} h_j$
4. Update decoder state: Compute $s_i = f(s_{i-1}, y_{i-1}, c_i)$
5. Generate output: Predict the next token using $s_i$ (and often $c_i$ as well)

One subtle but important detail: attention is computed using the previous decoder state $s_{i-1}$, not the current state $s_i$. This makes sense because we need to know what the decoder is looking for before we can compute the context, and we need the context before we can update the decoder state.