Multilayer Perceptrons and Backpropagation › Summary

A multilayer perceptron composes affine transformations with nonlinear activations. The hidden layer creates a learned representation, and the output layer maps that representation to a prediction. For binary classification, a sigmoid output with binary cross-entropy gives a stable loss when written in terms of the logit as $\operatorname{logaddexp}(0,s)-ys$.

Backpropagation is the chain rule applied backward through the network. The gradient at the output logit is predicted probability minus label. The output weights receive the hidden activations multiplied by that gradient, and the hidden preactivations receive the output gradient multiplied by the output weights and the activation derivative. Matrix notation makes the implementation compact: matrix products express weighted sums, while outer and elementwise products combine local gradient factors.

We verified the gradients with finite differences, trained a one-hidden-layer network with mini-batches and L2 regularization, and compared its decision boundary with logistic regression. The MLP can represent curved boundaries that a linear model cannot, while introducing nonconvex optimization and additional architectural and training choices. Input scaling and initialization require care. The held-out comparison evaluates this fitted model on new examples, while the gradient check tests its derivative implementation.