Multilayer Perceptrons and Backpropagation

A logistic regression model using the original input features draws a linear decision boundary. In the noisy two-moons example below, a straight line cuts across the crescent-shaped groups. A multilayer perceptron (MLP) can represent a curved boundary by stacking affine transformations with nonlinear activation functions. The first layer learns a new representation of the inputs, and the final layer makes a prediction from that representation.

The central difficulty is training. The hidden-layer network has more parameters than the linear baseline, and its loss is generally nonconvex in the combined layer parameters. We fit it with iterative gradient updates. Backpropagation computes the loss gradient with respect to every weight and bias by applying the chain rule systematically to the computation graph.

In this chapter we build a one-hidden-layer network from primitive NumPy operations. We derive the forward pass, the stable binary cross-entropy loss, and every gradient used in training. We check the gradients with finite differences, train the network with mini-batches, and compare it with logistic regression on a noisy nonlinear classification task. The chapter assumes familiarity with gradient descent and numerical optimization and with train/test discipline from validation strategies and data leakage.