Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multilayer Perceptrons › Forward Pass Computation

Michael BrenndoerferApril 29, 20256 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 63Multilayer Perceptrons › Forward Pass Computation

Reading section 4 of 13 · Forward Pass Computation · 6 minSection 4 of 13 · 6 min

Section 4 of 13 · 6 min read

Multilayer Perceptrons › Forward Pass ComputationOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

With our notation established, we can now understand how an MLP converts an input into an output. This process, called the forward pass, is the heart of neural network computation. Think of it as a pipeline: data flows in one direction, from input through hidden layers to output, with each layer transforming the representation along the way.

The Two-Step Layer ComputationLink Copied

At each layer, the network performs two distinct operations that work together to create expressive transformations.

Step 1: Linear Transformation. First, we compute a weighted combination of inputs from the previous layer:

\mathbf{z}^{[l]} = \mathbf{W}^{[l]} \mathbf{a}^{[l-1]} + \mathbf{b}^{[l]}

where:

$\mathbf{z}^{[l]}$ : the pre-activation vector at layer $l$
$\mathbf{W}^{[l]}$ : the weight matrix with shape $(n^{[l]}, n^{[l-1]})$
$\mathbf{a}^{[l-1]}$ : the activations from the previous layer
$\mathbf{b}^{[l]}$ : the bias vector for layer $l$

The weight matrix determines how strongly each input neuron influences each output neuron. The bias shifts the output, allowing neurons to activate even when inputs are zero.

Step 2: Non-linear Activation. Next, we apply an activation function element-wise to introduce non-linearity:

\mathbf{a}^{[l]} = \sigma(\mathbf{z}^{[l]})

where:

$\mathbf{a}^{[l]}$ : the activation vector at layer $l$ , which becomes input to the next layer
$\sigma$ : the activation function applied element-wise

Without activation functions, stacking multiple linear transformations would collapse into a single linear transformation, no matter how many layers we add. As we established in the Activation Functions chapter, $\mathbf{W}_2(\mathbf{W}_1 \mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2$ simplifies to a single linear transformation. Activation functions prevent this collapse.

The Complete Forward PassLink Copied

For a network with $L$ layers, the forward pass chains these two-step computations together. Starting with the input $\mathbf{x}$ (which we treat as $\mathbf{a}^{[0]}$ ), we propagate through each layer:

\begin{aligned} \mathbf{z}^{[1]} &= \mathbf{W}^{[1]} \mathbf{x} + \mathbf{b}^{[1]}, \quad \mathbf{a}^{[1]} = \sigma(\mathbf{z}^{[1]}) \\ \mathbf{z}^{[2]} &= \mathbf{W}^{[2]} \mathbf{a}^{[1]} + \mathbf{b}^{[2]}, \quad \mathbf{a}^{[2]} = \sigma(\mathbf{z}^{[2]}) \\ &\vdots \\ \mathbf{z}^{[L]} &= \mathbf{W}^{[L]} \mathbf{a}^{[L-1]} + \mathbf{b}^{[L]}, \quad \hat{\mathbf{y}} = \mathbf{a}^{[L]} \end{aligned}

where:

$\hat{\mathbf{y}}$ : the network's final output prediction
The output layer activation is chosen based on task (sigmoid for binary classification, softmax for multiclass, identity for regression)

Why This Architecture WorksLink Copied

The power of this layered structure comes from composition. Each layer learns to detect increasingly abstract features:

Early layers learn simple patterns directly from the input (edges, basic character shapes, common word fragments)
Middle layers combine these simple patterns into more complex features (textures, object parts, phrases)
Later layers assemble these features into high-level concepts (objects, categories, semantic meanings)

This hierarchical learning is what gives depth its power. Deeper networks discover representations at multiple levels of abstraction, something a single layer cannot achieve regardless of its width.

Implementing the Forward PassLink Copied

Let's translate these mathematical concepts into code. We'll build a forward pass function from scratch using NumPy to see exactly how the computation flows.

In[9]:

Code

def relu(z):
    """ReLU activation function: max(0, z)"""
    return np.maximum(0, z)


def sigmoid(z):
    """Sigmoid activation function: 1 / (1 + e^(-z))"""
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

Out[10]:

Visualization

Line plot of ReLU showing zero for negative inputs and linear growth for positive inputs. — ReLU activation passes positive values unchanged and zeros out all negative inputs, creating sparse activations that can speed up training.

Line plot of sigmoid showing an S-curve mapping all inputs to the (0, 1) range. — Sigmoid smoothly squashes all values to the (0, 1) range, making it suitable for probability outputs in binary classification.

Now we implement the forward pass itself. The function iterates through each layer, applying the two-step computation: linear transformation followed by activation.

In[11]:

Code

def forward_pass(
    x, weights, biases, hidden_activation=relu, output_activation=sigmoid
):
    """
    Compute forward pass through the network.

    Args:
        x: Input vector of shape (n_features, 1) or (n_features,)
        weights: List of weight matrices
        biases: List of bias vectors
        hidden_activation: Activation function for hidden layers
        output_activation: Activation function for output layer

    Returns:
        activations: List of activations for each layer (including input)
        pre_activations: List of pre-activation values for each layer
    """
    a = x.reshape(-1, 1) if x.ndim == 1 else x
    activations = [a]
    pre_activations = []

    for l in range(len(weights)):
        z = weights[l] @ a + biases[l]
        pre_activations.append(z)
        if l == len(weights) - 1:
            a = output_activation(z)
        else:
            a = hidden_activation(z)
        activations.append(a)

    return activations, pre_activations

Tracing Through a Concrete ExampleLink Copied

To solidify our understanding, let's trace through the forward pass with actual numbers. We'll use the 2-4-3-1 network we defined earlier and pass a single input through it.

In[12]:

Code

# Single input example
x_input = np.array([[0.5], [0.8]])

# Run forward pass
activations, pre_activations = forward_pass(x_input, weights, biases)

Out[13]:

Console

Input x:
  Shape: (2, 1), values: [0.5 0.8]

Layer 1:
  z[1]: shape (4, 1), values: [0.0689, 0.7711, -0.1522, 0.7018]
  a[1]: shape (4, 1), values: [0.0689, 0.7711, 0.0000, 0.7018]

Layer 2:
  z[2]: shape (3, 1), values: [0.0296, -0.9267, -0.4093]
  a[2]: shape (3, 1), values: [0.0296, 0.0000, 0.0000]

Layer 3:
  z[3]: shape (1, 1), values: [0.0217]
  a[3]: shape (1, 1), values: [0.5054]

Final output: 0.5054

The forward pass transforms our 2D input through three layers, ultimately producing a single scalar output. Several key observations emerge from this trace:

ReLU's effect: Notice how negative pre-activation values become zero after ReLU. This sparsity helps prevent overfitting and makes computations more efficient.
Dimensional changes: The representation changes size at each layer, from 2 dimensions (input) to 4, then 3, then finally 1 (output). The network progressively compresses information.
Sigmoid's bounded output: The final layer's sigmoid activation squashes the output to a probability between 0 and 1.
Composition creates complexity: Although each individual step is simple (matrix multiply, add bias, apply activation), the composition of many such steps creates a highly non-linear function.