Multilayer Perceptrons and Backpropagation › Visualizing Multilayer Perceptrons

The training curve shows the regularized full-training-set objective after each mini-batch epoch. Since updates use different mini-batches, this monitored objective need not decrease at every epoch. The displayed decrease describes the optimization run; judging model capacity and stopping time also requires validation results.

The next two figures compare the decision boundaries of logistic regression and the trained MLP on the same standardized test data. The first shows the logistic boundary, which is constrained to be a straight line in the input space. The second shows the MLP, which can bend its boundary because the hidden layer transforms the inputs before the output layer makes its decision. On a two-moons problem, this flexibility is exactly what allows the network to represent a curved decision region.

The decision-boundary figures build a probability grid over the feature space for both models before plotting, so the plotting cells below contain presentation logic only. The logistic panel shows the strong linear constraint of logistic regression, while the MLP panel shows a curved boundary that can bend to follow the crescent shapes.

The comparison illustrates the role of the hidden layer. The MLP boundary bends to follow the two moons more closely than the linear baseline does, which is consistent with the hidden layer providing the representational flexibility to represent a curved region on this task. This graphical observation is distinct from the earlier single-split test metrics, and it does not by itself establish that the MLP has better generalization performance than the linear baseline. The logistic model has $d+1$ parameters and a linear log-odds surface. The MLP has $h(d+1)+h+1$ parameters and can represent a much richer family of surfaces. That added capacity is not automatically better: with too little data, too much noise, or too little regularization, the MLP can fit idiosyncrasies of the training set. A validation split can guide capacity and regularization choices when tuning; that selection step is not implemented in this example, which uses prespecified hyperparameters.