Gradient Descent and Numerical Optimization › Disadvantages

Gradient descent uses local information. On a nonconvex objective, a small gradient can occur at a local minimum, a saddle, or a local maximum. A run may also diverge or reach its iteration limit. Its stopping status alone does not establish global optimality.

Curvature can make a stable run slow. For a positive-definite quadratic, the worst-case convergence rate depends on the Hessian condition number and the chosen learning rate. Large condition numbers make the best fixed-rate contraction approach one. The effect of rescaling features depends on their variances and correlations; it is not generally proportional to the change in their units.

We must also choose an update budget, stopping tolerances, and, for stochastic methods, a sampling rule and step schedule. Curvature sometimes provides a principled learning rate, while other problems need a line search or empirical tuning. Poor choices can cause divergence or stop a run before it reaches a useful fit.