Luong Attention › Worked Example: English-French Alignment
Language AI HandbookAll chapters Luong Attention › Worked Example: English-French AlignmentOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The formulas we've discussed can feel abstract until you see them in action. Let's trace through a complete attention computation for a concrete translation example, following each step from raw hidden states to final attention weights.
Consider translating the English sentence "The cat sat" to French "Le chat assis". We'll focus on the moment when the decoder is about to generate the second French word "chat". At this point, the decoder needs to figure out which English word to focus on. Intuitively, it should attend to "cat" since that's the word being translated.
Setting Up the Problem
First, let's create simulated encoder hidden states for each English word. In a real model, these would come from running a bidirectional LSTM or transformer encoder over the input. Here, we'll craft vectors that capture the intuition: "cat" and the decoder state for "chat" should be similar, while "The" and "sat" should be less relevant.
import numpy as np
import torch
np.random.seed(42)
torch.manual_seed(42)
# Create encoder representations that reflect semantic content:
# - "The": article with low semantic content (small magnitude)
# - "cat": noun with distinct pattern (will match decoder query)
# - "sat": verb with different pattern (won't match as well)
encoder_states = torch.tensor(
[
[0.1, 0.2, -0.1, 0.3], # "The" - low magnitude, generic
[0.8, -0.3, 0.9, 0.2], # "cat" - strong noun pattern
[0.2, 0.7, 0.1, -0.4], # "sat" - different verb pattern
],
dtype=torch.float32,
).unsqueeze(0) # Add batch dimension for PyTorch
# Decoder hidden state when generating "chat"
# Designed to be similar to "cat" encoding
decoder_state = torch.tensor(
[[0.7, -0.2, 0.8, 0.3]], # Query vector looking for noun-like content
dtype=torch.float32,
)Encoder states shape: torch.Size([1, 3, 4]) Decoder state shape: torch.Size([1, 4])
The encoder states have shape (1, 3, 4): batch size 1, 3 source words, and 4-dimensional hidden states. The decoder state has shape (1, 4): a single 4-dimensional vector representing what the decoder is "looking for" when generating "chat".
Step 1: Computing Alignment Scores
Now we apply the dot product score function. For each encoder state, we compute its dot product with the decoder state. The dot product measures how much two vectors point in the same direction.
# Compute dot product scores: decoder_state · each encoder_state
scores = torch.bmm(encoder_states, decoder_state.unsqueeze(2)).squeeze()
# Apply softmax to convert scores to probabilities
attention_weights = F.softmax(scores, dim=0)
# Compute context vector as weighted sum of encoder states
context = torch.bmm(
attention_weights.unsqueeze(0).unsqueeze(0), encoder_states
).squeeze()Step 2: Examining the Results
Let's see what scores each word received and how softmax converts them into attention weights:
Step-by-step attention computation: ---------------------------------------- Dot product scores (raw alignment): The: 0.0400 cat: 1.4000 sat: -0.0400 Attention weights (after softmax): The: 0.1718 cat: 0.6695 sat: 0.1586 Context vector: [0.585, -0.055, 0.601, 0.122] Highest attention on: 'cat'
The results reveal exactly what we hoped to see. The decoder state for generating "chat" produces the highest alignment score with "cat" because their vectors point in similar directions. The softmax function then amplifies this difference: "cat" receives about 70% of the attention weight, while "The" and "sat" share the remaining 30%.
Notice how the context vector ends up being dominated by the "cat" representation. This is attention in action: the decoder has learned to extract the relevant information from the source sequence by focusing on the semantically corresponding word.
The visualization below shows how the context vector is constructed as a weighted blend of the encoder states. Each dimension of the context vector is a weighted sum of that dimension across all encoder states, with the weights determined by attention.
