Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Feed-Forward Networks › A Complete Worked Example

Michael BrenndoerferUpdated June 10, 20257 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 107Feed-Forward Networks › A Complete Worked Example

Reading section 8 of 13 · A Complete Worked Example · 7 minSection 8 of 13 · 7 min

Section 8 of 13 · 7 min read

Feed-Forward Networks › A Complete Worked ExampleOne section · Sections in this chapter8 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The formula $\text{FFN}(x) = \sigma(xW_1 + b_1)W_2 + b_2$ is compact, but its compactness can obscure what happens at each step. To understand the FFN, let's trace through a concrete computation with actual numbers. We'll use intentionally small dimensions ( $d_{\text{model}} = 3$ and $d_{ff} = 4$ ) so you can follow every multiplication and addition by hand if you like, verifying each step against the formulas.

The goal of this example is threefold: to see exactly how the input vector is transformed at each stage, to observe how ReLU zeros out negative activations to introduce nonlinearity, and to verify that the output dimension matches the input dimension, ready for the residual connection. We'll use simple, hand-readable weight values to keep the arithmetic transparent.

In[22]:

Code

# Small example for hand-traceable computation
d_model_tiny = 3
d_ff_tiny = 4

# Initialize weights with simple values
W1_tiny = np.array(
    [
        [0.5, -0.3, 0.8, 0.2],
        [-0.2, 0.6, 0.1, -0.4],
        [0.3, 0.1, -0.5, 0.7],
    ]
)  # Shape: (3, 4)

b1_tiny = np.array([0.1, -0.1, 0.2, 0.0])  # Shape: (4,)

W2_tiny = np.array(
    [
        [0.4, -0.2, 0.3],
        [0.1, 0.5, -0.1],
        [-0.3, 0.2, 0.4],
        [0.2, -0.4, 0.1],
    ]
)  # Shape: (4, 3)

b2_tiny = np.array([0.05, -0.05, 0.1])  # Shape: (3,)

# Input vector
x_tiny = np.array([1.0, -0.5, 0.8])

We have chosen an input with mixed positive and negative values ( $x = [1.0, -0.5, 0.8]$ ), which is representative of what you'd see in practice after layer normalization. The weight matrices use a mix of positive and negative values. This ensures some hidden dimensions will have positive pre-activations (and survive ReLU) while others will be zeroed out.

Stage 1: Expansion via the First Linear LayerLink Copied

The first operation computes $xW_1 + b_1$ . Our 3-dimensional input vector gets multiplied by a $3 \times 4$ weight matrix, producing a 4-dimensional hidden representation. Each element of this output is a weighted sum of the input elements plus a bias term. Specifically, the $j$ -th element of the pre-activation is $x[0] \cdot W_1[0,j] + x[1] \cdot W_1[1,j] + x[2] \cdot W_1[2,j] + b_1[j]$ : a dot product between the input and the $j$ -th column of $W_1$ .

Out[23]:

Console

Step 1: First linear transformation (x @ W1 + b1)

Input x: [ 1.  -0.5  0.8]

W1:
[[ 0.5 -0.3  0.8  0.2]
 [-0.2  0.6  0.1 -0.4]
 [ 0.3  0.1 -0.5  0.7]]

b1: [ 0.1 -0.1  0.2  0. ]

x @ W1 + b1 = [ 0.94 -0.62  0.55  0.96]

Notice the dimension change: a 3-dimensional vector goes in, and a 4-dimensional vector comes out. This expansion is the "higher-dimensional working space" we discussed earlier, where the network has more room to manipulate the representation before applying nonlinearity. The result is called the pre-activation because we haven't applied the nonlinearity yet. Some of these pre-activation values are positive and some are negative, which sets up the selective filtering that ReLU will perform in the next step.

Stage 2: Nonlinearity via ReLULink Copied

Here is where the FFN gains its expressive power. The ReLU activation function applies a simple rule: keep positive values unchanged, but set negative values to zero. Mathematically, $\text{ReLU}(z) = \max(0, z)$ . This is one of the simplest possible nonlinear functions, yet it is sufficient to make the FFN a universal approximator when the hidden dimension is large enough.

Out[24]:

Console


Step 2: Apply ReLU activation

Pre-activation: [ 0.94 -0.62  0.55  0.96]
After ReLU:     [0.94 0.   0.55 0.96]

Negative values become zero, positive values pass through unchanged.

By zeroing out some dimensions, ReLU creates sparse hidden representations: only a subset of hidden dimensions are active for any given input. Different inputs activate different subsets, allowing the network to learn piece-wise linear functions that approximate arbitrary curves. Each hidden dimension can be thought of as a detector for a particular pattern in the input. When the detector fires (positive pre-activation), it contributes to the output. When it doesn't fire (negative pre-activation, zeroed by ReLU), it contributes nothing. The network learns during training which patterns each detector should respond to, and what contribution to make when it does respond.

The key insight here is that ReLU's behavior changes based on which side of zero the pre-activation falls. For pre-activations far above zero, the gradient of ReLU is 1 and learning proceeds normally. For pre-activations below zero, the gradient is 0 and the network cannot update that hidden dimension for that particular input. This creates a form of implicit regularization: hidden dimensions that consistently fail to fire for a category of inputs effectively "opt out" of processing that category, allowing specialization.

Out[25]:

Visualization

Bar chart showing pre-activation values with some negative (red) and some positive (blue). — Pre-activation values before ReLU. Negative values (red bars) indicate hidden dimensions where the input pattern did not match the corresponding key. These values will be zeroed out by ReLU.

Bar chart showing values after ReLU with negative values now at zero. — After ReLU: the two negative pre-activations (shown as red bars at zero) have been suppressed. Only the positive activations (blue bars) contribute to the final output, creating the sparse hidden representation characteristic of ReLU-based FFNs.

Stage 3: Contraction via the Second Linear LayerLink Copied

Finally, we project back to the original dimension. The 4-dimensional hidden vector is multiplied by a $4 \times 3$ weight matrix, and a 3-dimensional bias is added. This contraction reduces the dimension and combines the contributions of all active hidden dimensions into an output vector. Each output dimension receives a weighted sum of the active hidden dimensions, where the weights are the corresponding elements of $W_2$ .

Think of the second linear layer as the "summarization" step. The first layer asked "which features of this input are relevant?" The nonlinearity filtered out irrelevant dimensions. The second layer asks "given these relevant features, what transformation should we apply to the representation?"

Out[26]:

Console


Step 3: Second linear transformation (hidden @ W2 + b2)

Hidden: [0.94 0.   0.55 0.96]

W2:
[[ 0.4 -0.2  0.3]
 [ 0.1  0.5 -0.1]
 [-0.3  0.2  0.4]
 [ 0.2 -0.4  0.1]]

b2: [ 0.05 -0.05  0.1 ]

hidden @ W2 + b2 = [ 0.453 -0.512  0.698]

The output has the same dimension as the input (3 elements), which is essential. In the full transformer, this output will be added to the original input via a residual connection, and both must have the same shape. The FFN has transformed the representation while preserving its dimensionality, enabling the residual addition that stabilizes training in deep networks.

Verification and SummaryLink Copied

Let's verify that our step-by-step calculation matches the complete FFN function applied directly:

Out[27]:

Console


Verification using ffn() function:
  Output: [ 0.453 -0.512  0.698]
  Match: True

Summary:
  Input:  [ 1.  -0.5  0.8] (dimension 3)
  Output: [ 0.453 -0.512  0.698] (dimension 3)

The step-by-step and function-based computations match exactly. This worked example demonstrates the complete journey: a 3-dimensional input expands to 4 dimensions, passes through nonlinearity (with some dimensions zeroed out), and contracts back to 3 dimensions. The transformation is nonlinear (different inputs will activate different subsets of hidden dimensions, producing qualitatively different transformations), position-independent (the same computation would apply to any token's representation), and dimension-preserving (the output shape matches the input shape for the residual connection).