Gradient Descent and Numerical Optimization › Summary

Gradient descent uses the gradient to construct a local update. On a positive-definite quadratic, eigenvalue analysis determines the stable interval and the best worst-case fixed-rate contraction. The scalar recurrence $\theta_k=(1-\eta a)^k\theta_0$ makes convergence, sign alternation and divergence explicit. These exact statements require the stated quadratic conditions; a general nonconvex objective needs additional analysis.

The examples also show why the comparison setup matters. One common absolute rate gave the two quadratics identical shallow-coordinate progress; applying the same reciprocal-curvature rate rule separately exposed a much faster well-conditioned run. In least squares, keeping an explicit intercept made standardized and unscaled reference fits equivalent, while their finite-budget gradient-descent progress differed. Newton reached the quadratic solution to the reported numerical precision with one update.

Batch and stochastic variants use different amounts of data per update and require explicit sampling and work-count conventions. A numerical gradient check can catch implementation mistakes, and stopping diagnostics describe what a run achieved. Curvature, feature coordinates, numerical arithmetic and gradient noise all belong in that interpretation; a termination message alone is not a certificate of global optimality.