Multilayer Perceptrons and Backpropagation › Disadvantages

MLPs are nonconvex. The loss surface can contain many local minima, flat regions, saddle points, and steep cliffs. Gradient descent does not come with a global convergence guarantee. In practice, careful initialization, input scaling, sensible learning rates, and mini-batch noise make training work well on many problems, but the optimization is still more fragile than fitting a linear model or a convex regularized model.

MLPs are sensitive to preprocessing and hyperparameters. Features measured on very different scales can produce gradients with very different magnitudes, making a single learning rate awkward to choose. Training can stall or diverge. Hidden width, activation, learning rate, batch size, regularization strength, and training duration all affect the fitted model. A small network can underfit nonlinear structure; a large network can overfit, especially with few labeled examples.

Finally, MLPs are less interpretable than linear models. The weights of the first layer are not directly the effects of the original features, because they are mixed by the hidden units and the activation function. Interpretation usually requires post hoc methods, which we treat later in the book. For small tabular datasets where a linear model or a tree ensemble already performs well, the extra complexity of an MLP may not be justified.