Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Bahdanau Attention › A Worked Example: Attention in Action

Michael BrenndoerferUpdated May 22, 20256 min read

Language AI HandbookAll chapters

Part 12 of 67Sequence-to-Sequence

Chapter 87Bahdanau Attention › A Worked Example: Attention in Action

Reading section 4 of 11 · A Worked Example: Attention in Action · 6 minSection 4 of 11 · 6 min

Section 4 of 11 · 6 min read

Bahdanau Attention › A Worked Example: Attention in ActionOne section · Sections in this chapter4 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The formulas we've developed might seem abstract, so let's ground them with a concrete numerical example. We'll trace through every step of the attention computation, watching the numbers flow through each transformation.

Imagine we're building a translation system for English to French. We're translating "I love cats" and have just generated "J'" (the French equivalent of "I"). Now we need to generate the next word, which should be "aime" (love). The question is: which part of the input should the decoder focus on?

For this example, we'll use deliberately small dimensions so we can see every number:

Hidden dimension: 4 (real systems use 256-1024)
Alignment dimension: 3
Source sequence: 3 words ("I", "love", "cats")

In[13]:

Code

# Example dimensions for illustration
hidden_dim = 4
align_dim = 3
num_encoder_positions = 3

# Simulated encoder hidden states (normally from bidirectional RNN)
np.random.seed(42)
h = np.array(
    [
        [0.2, -0.5, 0.8, 0.1],  # h_1: "I"
        [0.9, 0.3, -0.2, 0.7],  # h_2: "love"
        [-0.1, 0.6, 0.4, -0.3],  # h_3: "cats"
    ]
)

# Previous decoder state (looking for "love" -> "aime")
s_prev = np.array([0.5, 0.1, -0.3, 0.8])

# Learnable parameters (normally trained)
W_a = np.array(
    [
        [0.3, -0.2, 0.5, 0.1],
        [0.4, 0.6, -0.1, 0.2],
        [-0.2, 0.3, 0.4, -0.5],
    ]
)  # Shape: (align_dim, hidden_dim)

U_a = np.array(
    [
        [0.2, 0.4, -0.3, 0.1],
        [-0.1, 0.5, 0.2, 0.3],
        [0.6, -0.2, 0.1, 0.4],
    ]
)  # Shape: (align_dim, hidden_dim)

v_a = np.array([0.5, -0.3, 0.4])  # Shape: (align_dim,)

Each encoder hidden state $h_j$ captures information about word $j$ and its surrounding context. The decoder state $s_{i-1}$ encodes what the decoder has generated so far and, implicitly, what it's looking for next. The weight matrices $W_a$ , $U_a$ , and vector $v_a$ would normally be learned during training; here we use fixed values for illustration.

Step 1: Project the Decoder StateLink Copied

The first step is to transform the decoder state into the alignment space. This projection happens once per decoder step and will be reused when comparing against each encoder position:

In[14]:

Code

# Project decoder state: W_a @ s_prev
W_s = W_a @ s_prev

Out[15]:

Console

W_a @ s_prev = [ 0.06  0.45 -0.59]

This 3-dimensional vector represents "what the decoder is looking for" in the alignment space. Think of it as a query that we'll compare against each encoder position.

Step 2: Compute Alignment ScoresLink Copied

Now we compute a score for each encoder position. For each position $j$ , we project $h_j$ into the alignment space, add it to the projected decoder state, apply tanh, and compute the final score with $v_a$ :

In[16]:

Code

scores = []
for j in range(num_encoder_positions):
    # Project encoder state
    U_h = U_a @ h[j]

    # Combine and apply tanh
    combined = np.tanh(W_s + U_h)

    # Compute scalar score
    e_j = v_a @ combined
    scores.append(e_j)

scores = np.array(scores)

Out[17]:

Console

Alignment scores:
  e_1 = -0.3634
  e_2 = 0.1092
  e_3 = -0.4023

The scores reveal that position 2 ("love") has the highest alignment with what the decoder is looking for. This makes intuitive sense: to generate "aime" (the French word for love), the decoder should focus on the English word "love." Position 3 ("cats") has the lowest score, indicating it's least relevant for this particular generation step.

Step 3: Normalize with SoftmaxLink Copied

Raw scores can be any real number, but we need probabilities. Softmax converts these scores into a proper probability distribution:

In[18]:

Code

# Softmax normalization
exp_scores = np.exp(scores)
attention_weights = exp_scores / exp_scores.sum()

Out[19]:

Console

Attention weights:
  α_1 = 0.2804
  α_2 = 0.4499
  α_3 = 0.2697
Sum: 1.0000

Position 2 receives about 45% of the attention, while positions 1 and 3 receive roughly 28% and 27% respectively. Notice that even though position 2 had the highest score, the attention isn't completely focused on it. This "soft" attention allows the model to hedge its bets, pulling in information from multiple positions when useful. The weights sum to exactly 1.0, confirming we have a valid probability distribution.

Step 4: Compute the Context VectorLink Copied

Finally, we use these attention weights to create a weighted combination of the encoder hidden states:

In[20]:

Code

# Weighted sum of encoder states
context = np.zeros(hidden_dim)
for j in range(num_encoder_positions):
    context += attention_weights[j] * h[j]

Out[21]:

Console

Context vector c_i = [0.43398358 0.15657815 0.24225493 0.26202583]

For comparison:
  h_2 ("love") = [ 0.9  0.3 -0.2  0.7]
  Difference  = 0.3724 (mean abs)

The context vector is a blend of all three encoder states, weighted by their relevance. It's closest to $h_2$ (the representation of "love") since that position received the highest attention weight, but it also incorporates information from the other positions. This context vector now gets fed into the decoder RNN along with the previous output embedding to generate the next token.

We can measure how the context vector relates to the encoder states using cosine similarity:

Context vector similarity to encoder states. The context vector is geometrically closest to $h_2$ ("love"), which received the highest attention weight (0.45).
Encoder State	Word	Attention Weight	Cosine Similarity
$h_1$	"I"	0.28	0.404
$h_2$	"love"	0.45	0.821
$h_3$	"cats"	0.27	0.150

The similarity analysis confirms that the context vector is closest to the encoder state that received the highest attention weight. However, the context isn't identical to any single encoder state; it's a weighted blend that can capture information from multiple positions when needed.

This worked example illustrates the key insight of attention: instead of forcing the decoder to work with a single fixed representation of the entire input, we dynamically construct a context that's tailored to each generation step. When generating "aime," the context emphasizes "love"; when generating "chats" later, the context would shift to emphasize "cats."