Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

RNN Architecture › A Worked Example: Single Forward Pass

Michael BrenndoerferMay 7, 20253 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 74RNN Architecture › A Worked Example: Single Forward Pass

Reading section 11 of 19 · A Worked Example: Single Forward Pass · 3 minSection 11 of 19 · 3 min

Section 11 of 19 · 3 min read

RNN Architecture › A Worked Example: Single Forward PassOne section · Sections in this chapter11 / 19 ↓

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let us trace through one complete forward pass of an RNN to see the mechanics concretely. We will use a tiny vocabulary and tiny hidden size to keep the numbers tractable.

Consider a vocabulary $\{"a", "b", "c"\}$ with one-hot encodings, a hidden size $H = 2$ , and the sequence "a", "b". We initialize $\mathbf{h}_0 = \mathbf{0}$ .

In[6]:

Code

import numpy as np

# Tiny RNN: D=3 (one-hot vocab), H=2 hidden units, V=3 output
D, H, V = 3, 2, 3

# Small, fixed weights for illustration
W_xh = np.array([[0.3, -0.2, 0.1], [-0.1, 0.4, 0.2]])  # shape (H, D)
W_hh = np.array([[0.5, 0.1], [-0.3, 0.6]])  # shape (H, H)
b_h = np.zeros(H)  # shape (H,)

W_hy = np.array([[0.2, -0.3], [0.1, 0.4], [-0.2, 0.1]])  # shape (V, H)
b_y = np.zeros(V)  # shape (V,)

# Input sequence: "a" = [1,0,0], "b" = [0,1,0]
x_a = np.array([1.0, 0.0, 0.0])
x_b = np.array([0.0, 1.0, 0.0])

# Step 1: process "a"
h0 = np.zeros(H)
h1 = np.tanh(W_hh @ h0 + W_xh @ x_a + b_h)

# Step 2: process "b"
h2 = np.tanh(W_hh @ h1 + W_xh @ x_b + b_h)

# Output at step 2 (logits)
y2_logits = W_hy @ h2 + b_y


# Softmax to get probabilities
def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()


y2_probs = softmax(y2_logits)

Out[7]:

Console

Forward pass through sequence ['a', 'b']
=============================================

h0 (initial):   [0. 0.]
h1 (after 'a'): [0.2913, -0.0997]
h2 (after 'b'): [-0.0642, 0.2476]

Output logits:  [-0.0871, 0.0926, 0.0376]
Output probs:   [0.3003, 0.3595, 0.3402]

Predicted next token: 'b'

Let us trace through the key computations step by step to understand exactly what is happening.

At step 1, the input $\mathbf{x}_a = [1, 0, 0]$ and the initial hidden state $\mathbf{h}_0 = [0, 0]$ . The hidden state update computes $\mathbf{W}_{hh}\,\mathbf{h}_0 + \mathbf{W}_{xh}\,\mathbf{x}_a + \mathbf{b}_h$ . Because $\mathbf{h}_0 = \mathbf{0}$ , the recurrent term vanishes and $\mathbf{h}_1 = \tanh(\mathbf{W}_{xh}[:,0] + \mathbf{0})$ , which is just the $\tanh$ of the first column of $\mathbf{W}_{xh}$ (corresponding to the "a" index). The hidden state after seeing "a" encodes the identity of "a" directly.

At step 2, the input is $\mathbf{x}_b = [0, 1, 0]$ and the previous hidden state is $\mathbf{h}_1$ (which depends on "a"). Now both terms contribute. The $\mathbf{W}_{hh}\,\mathbf{h}_1$ term brings forward information about "a", and the $\mathbf{W}_{xh}\,\mathbf{x}_b$ term injects information about "b". The combined pre-activation is a mixture of both, and $\tanh$ squashes the result. The hidden state $\mathbf{h}_2$ therefore encodes the fact that "b" followed "a", not just that the current token is "b".

This example uses only two tokens and two hidden units, but the same computation scales to sequences of thousands of tokens and hidden sizes of thousands of dimensions. The core insight is that the recurrent term in the hidden state update carries information from the past, blended with the current input by the weight matrices.

Visualizing Gradient Flow Through the Unrolled GraphLink Copied

It is illuminating to think about what gradient information flows back through this tiny example during training. The loss is computed at the output step (using $\mathbf{h}_2$ ). Backpropagating through $\mathbf{W}_{hy}$ gives gradients that adjust the output projection. Backpropagating through $\mathbf{h}_2$ requires propagating through the $\tanh$ and then through $\mathbf{W}_{hh}$ and $\mathbf{W}_{xh}$ to reach $\mathbf{h}_1$ . If the sequence were 10 steps long, the gradient from the loss at step 10 would need to propagate back through 10 applications of $\mathbf{W}_{hh}^T$ and 10 $\tanh$ derivative multiplications. This is where the vanishing gradient problem emerges.