Gradient Descent and Numerical Optimization

A decreasing loss can still leave the parameters far from their best values. A learning rate may make steady progress in one direction while barely moving in another; a larger rate can overshoot and diverge. On a quadratic objective, we can calculate these behaviors exactly and use them to check an implementation.

Gradient descent repeatedly measures the local slope and moves against it. For a differentiable objective with a nonzero gradient, a sufficiently small step in that direction decreases the objective. The gradient determines a direction; the learning rate determines the size of the displacement. Neither statement promises a global solution on an arbitrary nonconvex surface.

We will derive this update, compare two quadratics with controlled curvature, and contrast their paths with a Newton step from the same starting point. We will then fit a small least-squares model, check its gradient numerically, and examine batch, single-example, and mini-batch updates. The examples connect the objective from Sum of Squared Errors (SSE) with the coordinate changes introduced in standardization. The stopping rules will tell us what each run established.