Attention Intuition › Building Intuition with Code
Language AI HandbookAll chapters Attention Intuition › Building Intuition with CodeOne section · Sections in this chapter10 / 20 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The mathematics of attention translates directly into code. Let's implement the three-step process: compute alignment scores, apply softmax to get attention weights, and produce a context vector through weighted summation.
import numpy as np
np.random.seed(42)
# Simulate encoder outputs: 5 positions, each with 4-dimensional hidden state
encoder_states = np.random.randn(5, 4)
position_labels = ["The", "cat", "sat", "on", "mat"]
# Simulate decoder state (query): 4-dimensional
decoder_state = np.random.randn(4)Encoder states shape: (5, 4) Decoder state shape: (4,) Encoder states (each row is a position): The: [ 0.5 -0.14 0.65 1.52] cat: [-0.23 -0.23 1.58 0.77] sat: [-0.47 0.54 -0.46 -0.47] on: [ 0.24 -1.91 -1.72 -0.56] mat: [-1.01 0.31 -0.91 -1.41]
We have 5 encoder states representing the words in "The cat sat on mat". Each state is a 4-dimensional vector containing the hidden representation learned by the encoder. The decoder state, also 4-dimensional, represents what the model is currently trying to generate. In practice, these dimensions would be much larger (256 to 1024), but the small size here makes the computation easy to follow.
Now let's implement the attention mechanism. The function below follows our three-step formula exactly: compute dot product scores, apply softmax normalization, and return the weighted sum:
def compute_attention(query, keys, values):
"""
Compute attention weights and context vector using dot product scoring.
Args:
query: decoder state, shape (d,)
keys: encoder states used for scoring, shape (T, d)
values: encoder states to combine, shape (T, d)
Returns:
attention_weights: shape (T,)
context_vector: shape (d,)
"""
# Step 1: Compute alignment scores (dot product between query and each key)
scores = np.dot(keys, query)
# Step 2: Apply softmax to get attention weights (subtract max for numerical stability)
exp_scores = np.exp(scores - np.max(scores))
attention_weights = exp_scores / np.sum(exp_scores)
# Step 3: Compute context vector as weighted sum of values
context_vector = np.dot(attention_weights, values)
return attention_weights, context_vector
weights, context = compute_attention(
decoder_state, encoder_states, encoder_states
)Attention weights: The : 0.035 x cat : 0.039 x sat : 0.116 xxxx on : 0.604 xxxxxxxxxxxxxxxxxxxxxxxx mat : 0.206 xxxxxxxx Context vector: [-0.108 -1.042 -1.199 -0.601] Sum of weights: 1.000
The attention weights show how much the model focuses on each input position. Notice that the weights sum to exactly 1.0, forming a valid probability distribution over input positions. In this random example, the weights are distributed based on how similar each encoder state is to the decoder state (measured by dot product). The context vector is a weighted combination of all encoder states, with dimensions matching the encoder hidden size. In a trained model, these similarities would reflect learned relevance patterns rather than random correlations.
Let's also verify the three-step process explicitly, to make the connection between the formula and the code absolutely clear:
def compute_attention_verbose(query, keys, values, labels):
"""Same computation as compute_attention, with step-by-step printing."""
print(
"Step 1: Alignment scores (dot product of decoder state with each encoder state)"
)
scores = np.dot(keys, query)
for label, score in zip(labels, scores):
print(f" score({label}) = {score:.4f}")
print("\nStep 2: Attention weights (softmax of scores)")
exp_scores = np.exp(scores - np.max(scores))
attn_weights = exp_scores / np.sum(exp_scores)
for label, w in zip(labels, attn_weights):
print(f" alpha({label}) = {w:.4f}")
print(f" Sum of weights = {attn_weights.sum():.4f}")
print("\nStep 3: Context vector (weighted sum of encoder states)")
context = np.dot(attn_weights, values)
print(f" c = {context.round(4)}")
return attn_weights, contextStep 1: Alignment scores (dot product of decoder state with each encoder state) score(The) = -1.3670 score(cat) = -1.2771 score(sat) = -0.1783 score(on) = 1.4712 score(mat) = 0.3955 Step 2: Attention weights (softmax of scores) alpha(The) = 0.0354 alpha(cat) = 0.0387 alpha(sat) = 0.1160 alpha(on) = 0.6040 alpha(mat) = 0.2060 Sum of weights = 1.0000 Step 3: Context vector (weighted sum of encoder states) c = [-0.1085 -1.0418 -1.1986 -0.601 ]
This verbose version makes it easy to audit each step individually. Notice that the numerical stability trick (subtracting the maximum score before exponentiating) does not change the final weights; it prevents overflow errors when dealing with very large raw scores, which is especially important when scores can reach values like 20 or 30 in high-dimensional spaces.