Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multilayer Perceptrons › Network Architecture and Notation

Michael BrenndoerferApril 29, 20254 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 63Multilayer Perceptrons › Network Architecture and Notation

Reading section 3 of 13 · Network Architecture and Notation · 4 minSection 3 of 13 · 4 min

Section 3 of 13 · 4 min read

Multilayer Perceptrons › Network Architecture and NotationOne section · Sections in this chapter3 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

An MLP consists of an input layer, one or more hidden layers, and an output layer. We describe the architecture by the number of units in each layer. For example, a network with 4 inputs, two hidden layers of 8 and 4 units, and 2 outputs would be written as 4-8-4-2.

Let's establish notation that will serve us throughout this chapter and beyond:

$L$ : total number of layers (excluding input)
$n^{[l]}$ : number of neurons in layer $l$
$\mathbf{W}^{[l]}$ : weight matrix for layer $l$ , with shape $(n^{[l]}, n^{[l-1]})$
$\mathbf{b}^{[l]}$ : bias vector for layer $l$ , with shape $(n^{[l]}, 1)$
$\mathbf{z}^{[l]}$ : pre-activation values at layer $l$
$\mathbf{a}^{[l]}$ : activations (post-activation values) at layer $l$
$\mathbf{a}^{[0]} = \mathbf{x}$ : the input treated as the activation of layer 0

The weight matrix $\mathbf{W}^{[l]}$ connects layer $l-1$ to layer $l$ . Each row of $\mathbf{W}^{[l]}$ contains the weights for one neuron in layer $l$ . The element $W^{[l]}_{ij}$ represents the weight connecting neuron $j$ in layer $l-1$ to neuron $i$ in layer $l$ .

In[6]:

Code

# Define a simple 3-layer MLP: 2 inputs -> 4 hidden -> 3 hidden -> 1 output
layer_sizes = [2, 4, 3, 1]

# Initialize weight matrices and bias vectors
np.random.seed(42)
weights = []
biases = []

for l in range(1, len(layer_sizes)):
    W = np.random.randn(layer_sizes[l], layer_sizes[l - 1]) * 0.5
    b = np.zeros((layer_sizes[l], 1))
    weights.append(W)
    biases.append(b)

Out[7]:

Console

Network architecture: 2 -> 4 -> 3 -> 1

Weight matrix shapes:
  W[1]: (4, 2) (connects 2 neurons to 4 neurons)
  W[2]: (3, 4) (connects 4 neurons to 3 neurons)
  W[3]: (1, 3) (connects 3 neurons to 1 neurons)

Bias vector shapes:
  b[1]: (4, 1)
  b[2]: (3, 1)
  b[3]: (1, 1)

Total parameters: 31 (23 weights + 8 biases)

This 3-layer network has a modest parameter count, but the numbers grow quickly. A layer with 512 input neurons and 256 output neurons needs $512 \times 256 = 131{,}072$ weights alone. Let's visualize what these weight matrices look like:

Out[8]:

Visualization

Heatmap of weight matrix W1 with shape 4x2. — $W^{[1]}$: weight matrix connecting the 2-neuron input to the 4-neuron first hidden layer. Each row represents the incoming weights for one hidden neuron.

Heatmap of weight matrix W2 with shape 3x4. — $W^{[2]}$: weight matrix connecting first hidden (4 neurons) to second hidden layer (3 neurons). Color shows sign and magnitude of each connection.

Heatmap of weight matrix W3 with shape 1x3. — $W^{[3]}$: weight matrix connecting the 3-neuron second hidden layer to the single output neuron.

The weight matrices grow with the product of consecutive layer sizes. This rapid growth in parameters is why network architecture design requires careful consideration.