Multilayer Perceptrons › Batch Processing with Matrix Operations

Michael BrenndoerferApril 29, 20252 min read
Language AI HandbookAll chapters
Part 10 of 67Neural Network Foundations
Chapter 63Multilayer Perceptrons › Batch Processing with Matrix Operations
Reading section 5 of 13 · Batch Processing with Matrix Operations · 2 minSection 5 of 13 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 63 of 513

Section 5 of 13 · 2 min read

Multilayer Perceptrons › Batch Processing with Matrix OperationsOne section · Sections in this chapter5 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The forward pass we implemented processes one example at a time, but this is inefficient in practice. Modern hardware, especially GPUs, is designed for parallel computation. By processing multiple examples simultaneously, we can achieve dramatic speedups.

For a batch of mm examples, the input becomes a matrix X\mathbf{X} of shape (n[0],m)(n^{[0]}, m). The forward pass equations generalize elegantly:

Z[l]=W[l]A[l−1]+b[l]\mathbf{Z}^{[l]} = \mathbf{W}^{[l]} \mathbf{A}^{[l-1]} + \mathbf{b}^{[l]}

where:

  • A[l−1]\mathbf{A}^{[l-1]}: the activation matrix from the previous layer, with shape (n[l−1],m)(n^{[l-1]}, m)
  • W[l]\mathbf{W}^{[l]}: the weight matrix with shape (n[l],n[l−1])(n^{[l]}, n^{[l-1]})
  • Z[l]\mathbf{Z}^{[l]}: the pre-activation matrix with shape (n[l],m)(n^{[l]}, m)
  • b[l]\mathbf{b}^{[l]}: the bias vector with shape (n[l],1)(n^{[l]}, 1), broadcast across all mm columns
  • mm: the batch size

The matrix multiplication computes the linear transformation for all examples at once. The bias vector is broadcast (replicated) across all mm columns.

In[14]:
Code
def forward_pass_batch(
    X, weights, biases, hidden_activation=relu, output_activation=sigmoid
):
    """
    Compute forward pass for a batch of inputs.

    Args:
        X: Input matrix of shape (n_features, batch_size)
        weights: List of weight matrices
        biases: List of bias vectors

    Returns:
        activations: List of activation matrices for each layer
        pre_activations: List of pre-activation matrices for each layer
    """
    A = X if X.ndim == 2 else X.reshape(-1, 1)
    activations = [A]
    pre_activations = []

    for l in range(len(weights)):
        Z = weights[l] @ A + biases[l]
        pre_activations.append(Z)
        if l == len(weights) - 1:
            A = output_activation(Z)
        else:
            A = hidden_activation(Z)
        activations.append(A)

    return activations, pre_activations


# Create a batch of 5 examples
X_batch = np.random.randn(2, 5)
activations_batch, _ = forward_pass_batch(X_batch, weights, biases)
Out[15]:
Console
Batch input shape: (2, 5) (2 features, 5 examples)
Batch output shape: (1, 5) (1 output, 5 examples)

Outputs for each example:
  Example 1: 0.5000
  Example 2: 0.5000
  Example 3: 0.5000
  Example 4: 0.5528
  Example 5: 0.5000

All five examples are processed in a single matrix operation, producing five outputs simultaneously. Batch processing also provides more stable gradient estimates during training, as we will see in the Backpropagation chapter.