Gradient Descent and Numerical Optimization › Example

We can check the recurrence by hand, then compare its predicted iterates with the update loop.

Take the well-conditioned curvature $a = (2, 1)$, start at $\theta_0 = (1, 1)$, and use $\eta = 0.25$. The gradient at the start is $(2\cdot 1, 1\cdot 1) = (2, 1)$, so the first update is

The objective falls from $J(\theta_0) = \tfrac12(2\cdot 1^2 + 1\cdot 1^2) = \tfrac12(2 + 1) = 1.5$ to $J(\theta_1) = \tfrac12(2\cdot 0.5^2 + 1\cdot 0.75^2) = \tfrac12(0.5 + 0.5625) = \tfrac12(1.0625) = 0.53125$, where $\theta_0$ and $\theta_1$ are the parameters before and after the first update, each term is $\tfrac12 a_j \theta_{j}^2$ with $a_j$ the curvature and $\theta_j$ the parameter along coordinate $j$, and the intermediate sums make the arithmetic explicit. Continuing, we can factor the update per coordinate. For coordinate one the multiplier is $1 - \eta a_1 = 1 - 0.25 \cdot 2 = 1 - 0.5 = 0.5$; for coordinate two it is $1 - \eta a_2 = 1 - 0.25 \cdot 1 = 1 - 0.25 = 0.75$, where $\eta$ is the learning rate and $a_1, a_2$ are the curvatures along the two coordinates. So after $k$ steps

where $\theta_{1,k}$ and $\theta_{2,k}$ are the two coordinates at iteration $k$. With this starting point and rate, coordinate two decays more slowly. After ten steps it is $0.75^{10} \approx 0.0563$, while coordinate one is $0.5^{10} \approx 0.00098$. Here the smaller-curvature direction has the larger contraction multiplier. For another rate, we must compare both absolute multipliers rather than assuming that the smallest curvature always controls the error.

Now verify the Newton prediction. The Hessian is constant, so the step from $\theta_0$ is

since $H^{-1} = \mathrm{diag}(1/2, 1)$ and the gradient is $(2,1)$, where:

• $H$: the Hessian with diagonal entries equal to the curvatures $a_1 = 2$ and $a_2 = 1$
• $H^{-1}$: the inverse of the Hessian, here $\mathrm{diag}(1/2, 1)$
• $\nabla J(\theta_0)$: the gradient at the starting point, here $(2, 1)$
• $d$: the Newton step

Adding gives $\theta_1 = (1,1) + (-1,-1) = (0,0)$, the exact minimum, in a single step. Let us confirm these hand calculations with code so nothing rests on arithmetic we did in our heads.

The example also exposes a subtlety about learning rates. With $\eta = 0.25$ both coordinates contract, and the objective decreases monotonically. Raise $\eta$ to $0.75$ and coordinate one has multiplier $1 - \eta a_1 = 1 - 0.75 \cdot 2 = 1 - 1.5 = -0.5$, so it oscillates in sign while shrinking; the objective still decreases each step but the path zig-zags. The multiplier for coordinate $j$ is $1 - \eta a_j$, with $\eta$ the learning rate and $a_j$ the curvature along that coordinate. Raise $\eta$ to $1.25$ and the multiplier for coordinate one is $1 - 1.25 \cdot 2 = 1 - 2.5 = -1.5$, magnitude greater than one, and the run diverges even though coordinate two still converges. The next cell confirms the threshold numerically.

The run with $\eta = 1.25$ diverges because of the steep direction. On a positive-definite quadratic, an unbounded parameter eventually forces an unbounded objective. Over a finite budget, a lightly weighted coordinate can still look small in the objective while its parameter error matters. Inspect the parameter trajectory alongside the loss when that distinction is relevant.