Multilayer Perceptrons and Backpropagation › Example

To make the formula concrete, we compute it by hand. The following example is deliberately tiny: two input features, two hidden units, and one binary output. Its purpose is not to be realistic but to expose every number that the code will later compute. We will carry the forward pass, the backward pass, and one gradient step by hand, so that each matrix in the implementation has a concrete scalar counterpart. Calculations retain full precision; displayed results are rounded to four decimal places.

Suppose we have two input features, two hidden units, and one binary output. Let the input be

$$x = \begin{bmatrix}1.0\\-1.0\end{bmatrix}, \qquad y=1.$$

Let the hidden weights and biases be

$$\mathbf{W} = \begin{bmatrix} 0.5 & -0.5\\ 1.0 & 0.5 \end{bmatrix}, \qquad \mathbf{b}= \begin{bmatrix} 0.1\\-0.2 \end{bmatrix},$$

and let the output weights and bias be

$$\mathbf{v}= \begin{bmatrix} 1.2\\-0.8 \end{bmatrix}, \qquad c=0.05.$$

We use $\tanh$ in the hidden layer and sigmoid at the output.

We compute the hidden preactivations one unit at a time. For the first hidden unit,

$$z_1 = 0.5(1.0)+(-0.5)(-1.0)+0.1 = 0.5+0.5+0.1=1.1.$$

Then

$$a_1=\tanh(1.1)\approx 0.8005.$$

For the second hidden unit,

$$z_2 = 1.0(1.0)+0.5(-1.0)-0.2 = 1.0-0.5-0.2=0.3,$$

so

$$a_2=\tanh(0.3)\approx 0.2913.$$

The output logit is

$$s = 1.2a_1-0.8a_2+0.05\approx 0.7775.$$

The predicted probability is

$$p=\sigma(s)=\frac{1}{1+e^{-s}}\approx 0.6852.$$

where:

• $\sigma$: the sigmoid function mapping the logit to a probability
• $p$: the predicted probability that the example belongs to class $1$

The loss is

$$\ell = -\log(p)\approx 0.3781.$$

where:

• $\ell$: the binary cross-entropy loss for this single example
• $\log$: the natural logarithm

Equivalently, using the stable logit form,

$$\ell=\operatorname{logaddexp}(0,s)-s\approx 0.3781.$$

We can read this forward pass as a pipeline: the input is projected into two hidden numbers, each hidden number is squashed by tanh, and the two squashed numbers are combined into a logit. The predicted probability is above $0.5$, so the network already leans toward class $1$, and the observed label is $1$. Since the assigned probability $p\approx0.6852$ is below $1$, its binary cross-entropy $-\log p\approx0.3781$ is positive.

We now reverse the forward pass. The output gradient is

$$\delta = p-y \approx -0.3148,$$

The output weight gradients are

$$\frac{\partial \ell}{\partial v_1}=\delta a_1\approx -0.2520,$$ $$\frac{\partial \ell}{\partial v_2}=\delta a_2\approx -0.0917,$$

where:

• $\delta = p - y$: the output delta for this single example, without the $1/B$ scaling used for a batch mean
• $a_1, a_2$: the hidden activations computed in the forward pass
• $v_1, v_2$: the output weights

and the output bias gradient is

$$\frac{\partial \ell}{\partial c}=\delta\approx -0.3148.$$

For the hidden units, we first compute

$$\frac{\partial \ell}{\partial a_1}=\delta v_1\approx -0.3778,$$ $$\frac{\partial \ell}{\partial a_2}=\delta v_2\approx 0.2519.$$

The tanh derivatives are

$$1-a_1^2\approx 0.3592, \qquad 1-a_2^2\approx 0.9151,$$

Therefore the hidden preactivation gradients are

$$\Delta_1=\delta v_1(1-a_1^2)\approx -0.1357,$$ $$\Delta_2=\delta v_2(1-a_2^2)\approx 0.2305.$$

Now we can compute the weight gradients. Since $z_1=W_{11}x_1+W_{12}x_2+b_1$,

$$\frac{\partial \ell}{\partial W_{11}}=\Delta_1 x_1\approx -0.1357,$$ $$\frac{\partial \ell}{\partial W_{12}}=\Delta_1 x_2\approx 0.1357.$$

Similarly,

$$\frac{\partial \ell}{\partial W_{21}}=\Delta_2 x_1\approx 0.2305,$$ $$\frac{\partial \ell}{\partial W_{22}}=\Delta_2 x_2\approx -0.2305.$$

The hidden bias gradients are simply $\Delta_1$ and $\Delta_2$:

$$\frac{\partial \ell}{\partial b_1}\approx -0.1357, \qquad \frac{\partial \ell}{\partial b_2}\approx 0.2305.$$

At this point every gradient has been computed by following the chain rule one link at a time. The output delta is the derivative of the loss with respect to the logit; its sign tells us which local change in the logit would reduce the loss. Multiplying by the hidden activations tells us how to change the output weights. Multiplying the output delta by the output weights and tanh derivative gives the loss gradient with respect to each hidden preactivation. Finally, multiplying by the input tells us how to change the hidden weights.

Gradient descent subtracts a multiple of the gradient. With learning rate $\eta=0.5$, the new first hidden weight is

$$W_{11}^{\text{new}}=W_{11}-\eta\frac{\partial\ell}{\partial W_{11}}\approx 0.5679.$$

where:

• $W_{11}^{\text{new}}$: the updated hidden weight after the gradient step
• $0.5$: the initial value of $W_{11}$
• $0.5$: the learning rate $\eta$
• $-0.1357$: the gradient $\partial \ell/\partial W_{11}$

The new first output weight is

$$v_1^{\text{new}}=v_1-\eta\frac{\partial\ell}{\partial v_1}\approx 1.3260.$$

where:

• $v_1^{\text{new}}$: the updated first output weight after the gradient step
• $1.2$: the initial value of $v_1$
• $0.5$: the learning rate $\eta$
• $-0.2520$: the rounded gradient $\partial \ell/\partial v_1$

Applying the same rule at full precision and rounding the displayed values gives

$$\begin{aligned} W_{11}&\approx0.5679, & W_{12}&\approx-0.5679,\\ W_{21}&\approx0.8847, & W_{22}&\approx0.6153,\\ b_1&\approx0.1679, & b_2&\approx-0.3153,\\ v_1&\approx1.3260, & v_2&\approx-0.7541,\\ c&\approx0.2074. \end{aligned}$$

A new forward pass with these updated parameters gives $z_1\approx 1.3036$, $a_1\approx 0.8626$, $z_2\approx -0.0458$, $a_2\approx -0.0457$, $s\approx 1.3858$, $p\approx 0.7999$, and loss $\approx 0.2232$. The loss decreased from about $0.378$ to about $0.223$, which is the expected effect of a step in the negative gradient direction. It is a small confirmation that the signs and scales in the backward pass are consistent with the goal of reducing the loss.