Gradient Descent and Numerical Optimization

A decreasing loss can still leave the parameters far from their best values. A learning rate may make steady progress in one direction while barely moving in another; a larger rate can overshoot and diverge. On a quadratic objective, we can calculate these behaviors exactly and use them to check an implementation.

Gradient descent repeatedly measures the local slope and moves against it. For a differentiable objective with a nonzero gradient, a sufficiently small step in that direction decreases the objective. The gradient determines a direction; the learning rate determines the size of the displacement. Neither statement promises a global solution on an arbitrary nonconvex surface.

We will derive this update, compare two quadratics with controlled curvature, and contrast their paths with a Newton step from the same starting point. We will then fit a small least-squares model, check its gradient numerically, and examine batch, single-example, and mini-batch updates. The examples connect the objective from Sum of Squared Errors (SSE) with the coordinate changes introduced in standardization. The stopping rules will tell us what each run established.

The update needs an objective and its gradient rather than a closed-form minimizer. We can therefore use it for smooth fitting problems such as logistic regression, using negative log-likelihood as the objective. For an unconstrained problem, the same update structure applies when we change the loss and compute its new gradient. Nondifferentiable objectives require an appropriate alternative, such as a subgradient or proximal method.

For the dense linear model used here, a full gradient costs $O(np)$ operations for $n$ rows and $p$ parameters. A single-example or mini-batch update can use less data before changing the parameters. These first-order updates avoid forming a $p\times p$ Hessian, and data can be supplied in batches when storing the full design is inconvenient.

An iterative fit also produces a history that we can inspect. We can record the objective, gradient norm, parameter changes, and stopping reason. Computing these diagnostics has a cost, and an increasing objective can be expected during stochastic updates, so the appropriate interpretation depends on the algorithm.

Smooth penalties can be included in the objective and differentiated along with the data-fit term. Constraints need additional treatment, such as a projection or a solver designed for the feasible set; an unconstrained gradient step alone does not enforce them.

Gradient descent uses local information. On a nonconvex objective, a small gradient can occur at a local minimum, a saddle, or a local maximum. A run may also diverge or reach its iteration limit. Its stopping status alone does not establish global optimality.

Curvature can make a stable run slow. For a positive-definite quadratic, the worst-case convergence rate depends on the Hessian condition number and the chosen learning rate. Large condition numbers make the best fixed-rate contraction approach one. The effect of rescaling features depends on their variances and correlations; it is not generally proportional to the change in their units.

We must also choose an update budget, stopping tolerances, and, for stochastic methods, a sampling rule and step schedule. Curvature sometimes provides a principled learning rate, while other problems need a line search or empirical tuning. Poor choices can cause divergence or stop a run before it reaches a useful fit.

Let $\theta\in\mathbb{R}^p$ be the parameter vector and let $J(\theta)$ be the real-valued objective to minimize. We begin with a local approximation to $J$, then use quadratic examples to determine exactly when repeated updates converge.

The entries of $\theta$ are the quantities we fit. In the regression example they will be an intercept and slopes; the objective is half the mean squared prediction error. Unless stated otherwise, the derivations concern differentiable objectives on an unconstrained parameter space.

The first thing we need is a compact summary of how $J$ responds to a nudge in any of the $p$ coordinate directions. That summary is the gradient, which collects all first partial derivatives into one vector:

Read the vector one entry at a time and each entry asks the same question in a different coordinate: if we nudge only $\theta_j$ by a tiny amount and hold the other $p-1$ parameters fixed, how fast does $J$ change, and in which direction? Stacking those answers into a column vector turns $p$ separate one-dimensional sensitivities into a single object that encodes both the steepest direction and the steepness itself. Two facts about that object carry everything that follows:

1. For a nonzero gradient, its negative gives the steepest directional decrease among directions of unit Euclidean length. This is a first-order statement about a small displacement, not the best destination at an arbitrary finite distance.
2. The gradient norm measures the first-order residual in the chosen units. A differentiable unconstrained local minimum has zero gradient, but a small norm alone does not measure distance to a minimizer or certify that a point is a minimum.

A direction alone does not tell us how far to move. The learning rate supplies that scale.

Let $d$ be a direction vector and $t$ a scalar step multiplier. A first-order Taylor expansion around the current point says

The left side is the objective after moving by $td$. The right side adds a first-order prediction of that change to the current objective. The dot product $\nabla J(\theta)^{\top}d$ is the directional derivative along $d$. When both vectors are nonzero, it is negative if their angle exceeds ninety degrees. Among unit Euclidean directions, Cauchy-Schwarz gives the smallest dot product at $d=-\nabla J(\theta)/\|\nabla J(\theta)\|$, provided the gradient is nonzero. Absorbing the step length into a positive scalar gives the gradient-descent update

where:

• $\theta_k$: the parameter vector at iteration $k$
• $\theta_{k+1}$: the updated parameter vector after one step
• $\eta > 0$: the learning rate, controlling how far we travel along the negative gradient
• $\nabla J(\theta_k)$: the gradient of the objective evaluated at the current parameters, pointing in the direction of steepest increase
• $k$: the iteration counter

Reading it plainly: the new parameters are the old parameters, shifted a small distance against the gradient. In scalar form, component $j$ obeys $\theta_{k+1,j} = \theta_{k,j} - \eta \, \partial J/\partial \theta_j$, where $\theta_{k+1,j}$ is the $j$-th component after the update, $\theta_{k,j}$ is the $j$-th component before the update, $\eta$ is the learning rate, and $\partial J/\partial \theta_j$ is the partial derivative of the objective with respect to the $j$-th parameter. This component form is often easier to reason about when we implement it.

The displacement has Euclidean length $\eta\|\nabla J(\theta_k)\|$: the learning rate multiplies the gradient rather than directly specifying a distance. For example, when fitting a likelihood model we minimize its negative log-likelihood and differentiate that objective. A local linear approximation need not remain accurate over a large displacement, so the step length requires a separate choice.

For the scalar quadratic $J(\theta)=a\theta^2/2$ with $a>0$, the gradient is $a\theta$. Substituting it into the update gives

where:

• $\theta_k$: the parameter at iteration $k$
• $a > 0$: the curvature, the second derivative of $J$ with respect to $\theta$
• $\eta > 0$: the learning rate
• $(1 - \eta a)$: the signed multiplier of the scalar parameter at each update

The quantity $1 - \eta a$ multiplies the parameter anew at every iteration, which means after $k$ steps

where $\theta_0$ is the starting value at iteration zero.

For every starting value, convergence to zero occurs when $|1-\eta a|<1$, equivalently $0<\eta<2/a$. At $\eta=1/a$, a single update reaches zero. For a nonzero starting value, $\eta=2/a$ produces constant-amplitude sign changes, and $\eta>2/a$ produces divergence. A start already at zero stays there for any finite learning rate. These exact recursions use the constant curvature of this quadratic; a general smooth objective needs a separate local or global analysis.

The relevant step scale is reciprocal curvature. For $a>0$, larger curvature narrows the stable interval $0<\eta<2/a$. This explains why a rate that works on one quadratic may be unstable on another. It does not imply that every small rate is equally efficient.

For a quadratic with a constant symmetric positive-definite Hessian $H$, all eigenvalues satisfy $0<\lambda_{\min}\le\lambda_i\le\lambda_{\max}$. Write the error relative to its unique minimizer in the orthonormal eigenvector basis of $H$. Each error coordinate is multiplied by $1-\eta\lambda_i$, so the Euclidean error contracts by at most $\rho(\eta)=\max_i|1-\eta\lambda_i|$. Convergence from every start requires $0<\eta<2/\lambda_{\max}$. Balancing the two extreme multipliers gives the optimal fixed rate $\eta=2/(\lambda_{\min}+\lambda_{\max})$ and its worst-case parameter-error contraction factor

where:

• $\kappa$: the condition number of the Hessian, defined as $\lambda_{\max}/\lambda_{\min}$, which measures how differently the objective curves across directions
• $\lambda_{\min}, \lambda_{\max}$: the smallest and largest eigenvalues of the Hessian $H$, the curvatures along the shallowest and steepest directions respectively
• $(\kappa - 1)/(\kappa + 1)$: the worst-case per-iteration error contraction factor for the optimal fixed learning rate

When $\kappa=1$, this quadratic has equal curvature in every direction and the optimal fixed rate reaches the minimizer in one update. At $\kappa=100$, the factor is $99/101\approx0.980198$; the worst-case parameter error needs 116 updates for a tenfold reduction because $(99/101)^{116}<0.1$. This comparison uses relative parameter error and the optimal fixed rate. Objective gaps along an individual eigenmode contract by the square of that mode's parameter multiplier.

For $\mathrm{SSE}(\theta)=\|y-X\theta\|^2$, differentiation gives gradient $-2X^{\top}(y-X\theta)$ and Hessian $2X^{\top}X$. Here $X$ has $n$ rows and $p$ columns, $y$ has $n$ entries, and $\theta$ has $p$ entries. If $X$ has full column rank, the Hessian is positive definite and its spectral condition number equals the square of the design matrix's spectral condition number. Rescaling feature columns changes this geometry. Standardization often helps when unequal feature scales cause poor conditioning, but it does not remove collinearity or guarantee fast convergence. With rank deficiency, the Hessian is singular and the positive-condition-number formulas above do not apply.

When $J(\theta)=n^{-1}\sum_{i=1}^n\ell_i(\theta)$ is a finite mean of differentiable example losses, differentiation gives $\nabla J(\theta)=n^{-1}\sum_i\nabla\ell_i(\theta)$. The full-batch method computes this complete average before an update. Its work is $n$ times the cost of one example gradient; for the dense linear model here that is $O(np)$.

A single-example update uses one loss gradient. If a fresh index $i$ is sampled uniformly and independently of previous updates, its gradient at the current fixed parameters has conditional expectation equal to the full gradient. The update is

A single-example gradient can vary around the full gradient, so individual updates need not decrease the full objective. Persistent noise near an optimum can leave fluctuations when the step stays constant. Positive schedules satisfying $\eta_k\to0$, $\sum_k\eta_k=\infty$, and $\sum_k\eta_k^2<\infty$ are used in convergence analyses together with assumptions on the objective and noise. A smooth strongly convex objective, conditionally unbiased bounded-variance gradients, and stable iterates are one such setting; the schedule alone is not a guarantee for an arbitrary objective. Noise can also vanish at the solution, in which case convergence with a constant step is possible. Our implementation uses random reshuffling, whose adaptive updates are not automatically covered by the fresh-index unbiasedness calculation.

Mini-batch descent sits between the extremes, using a batch $B_k$ of size $b$:

At fixed parameters, averaging $b$ independent example gradients divides their covariance by $b$. Sampling a subset without replacement has a different finite-population factor and gives zero sampling variance when the batch contains all $n$ examples. Batch size also changes the number of updates per pass and the available vectorization. We use 32 as one illustrative mini-batch size below; it is not a universal optimum.

If curvature is the problem, we can use it. Rather than guessing a step size that survives the worst direction, expand the objective to second order around the current point:

The Hessian $H(\theta)$ is the matrix of second partial derivatives. The new quadratic term models how the slope changes over the displacement $d$. When this Hessian is positive definite, the quadratic approximation has a unique minimizer. Differentiating that approximation with respect to $d$ and setting its gradient to zero gives

So the Newton update is

The displayed rule is the full Newton step, with step multiplier one. In practice we solve $H(\theta_k)d=-\nabla J(\theta_k)$ rather than explicitly forming an inverse. On a positive-definite quadratic this reaches the minimizer in one update in exact arithmetic. Local quadratic convergence for a nonquadratic objective is guaranteed under a sufficiently close start, a nonsingular Hessian at the solution, and a Hessian that is locally Lipschitz, meaning its changes are bounded proportionally to parameter displacement nearby. These sufficient conditions are stronger than smoothness alone.

A dense direct solve typically costs $O(p^3)$ arithmetic and stores a $p\times p$ matrix; forming the Hessian has an additional cost. Newton-type methods can also use iterative solves or Hessian-vector products, so explicit dense storage is not universal. An indefinite Hessian can produce an uphill direction. A shift must be large enough to make the modified Hessian positive definite; at a nonzero gradient that provides a descent direction. A line search can then shorten a descent step to achieve sufficient decrease. Shortening an uphill direction alone does not repair its sign.

For stopping, we need honest criteria rather than a fixed iteration count. Three are standard, and they answer different questions:

• Absolute gradient norm $\|\nabla J(\theta_k)\| < \epsilon_g$ checks an approximate first-order stationarity condition in the current units.
• Relative objective change $|J(\theta_k) - J(\theta_{k-1})| / \max(1, |J(\theta_{k-1})|) < \epsilon_J$ asks whether progress has stalled.
• Relative parameter change $\|\theta_k - \theta_{k-1}\| / \max(1, \|\theta_{k-1}\|) < \epsilon_\theta$ asks whether the iterates have settled.

Each can misfire alone. A tiny objective change may reflect a flat but nonoptimal plateau, and a small gradient norm may occur at a saddle. Report more than one diagnostic alongside the objective value at the returned point. A finite-difference estimate provides a separate check on a hand-derived gradient: it can expose a sign, normalization, or indexing mistake before we interpret the training curve.

We use two diagonal quadratic objectives with known minima to inspect the update paths. For a two-entry parameter vector, define

The positive coefficients $a_1$ and $a_2$ are the curvatures along the coordinate axes. The gradient is $(a_1\theta_1,a_2\theta_2)$, the Hessian is $\mathrm{diag}(a_1,a_2)$, and the unique minimizer is the origin. Here $a_1\ge a_2>0$, so the condition number is $a_1/a_2$.

The displayed Newton solution has zero objective and gradient to the reported numerical precision. The code solves a linear system with the exact diagonal Hessian. This example confirms the one-step algebra for this positive-definite quadratic; it does not establish one-step convergence for arbitrary objectives or immunity to floating-point conditioning.

We can check the recurrence by hand, then compare its predicted iterates with the update loop.

Take the well-conditioned curvature $a = (2, 1)$, start at $\theta_0 = (1, 1)$, and use $\eta = 0.25$. The gradient at the start is $(2\cdot 1, 1\cdot 1) = (2, 1)$, so the first update is

The objective falls from $J(\theta_0) = \tfrac12(2\cdot 1^2 + 1\cdot 1^2) = \tfrac12(2 + 1) = 1.5$ to $J(\theta_1) = \tfrac12(2\cdot 0.5^2 + 1\cdot 0.75^2) = \tfrac12(0.5 + 0.5625) = \tfrac12(1.0625) = 0.53125$, where $\theta_0$ and $\theta_1$ are the parameters before and after the first update, each term is $\tfrac12 a_j \theta_{j}^2$ with $a_j$ the curvature and $\theta_j$ the parameter along coordinate $j$, and the intermediate sums make the arithmetic explicit. Continuing, we can factor the update per coordinate. For coordinate one the multiplier is $1 - \eta a_1 = 1 - 0.25 \cdot 2 = 1 - 0.5 = 0.5$; for coordinate two it is $1 - \eta a_2 = 1 - 0.25 \cdot 1 = 1 - 0.25 = 0.75$, where $\eta$ is the learning rate and $a_1, a_2$ are the curvatures along the two coordinates. So after $k$ steps

where $\theta_{1,k}$ and $\theta_{2,k}$ are the two coordinates at iteration $k$. With this starting point and rate, coordinate two decays more slowly. After ten steps it is $0.75^{10} \approx 0.0563$, while coordinate one is $0.5^{10} \approx 0.00098$. Here the smaller-curvature direction has the larger contraction multiplier. For another rate, we must compare both absolute multipliers rather than assuming that the smallest curvature always controls the error.

Now verify the Newton prediction. The Hessian is constant, so the step from $\theta_0$ is

since $H^{-1} = \mathrm{diag}(1/2, 1)$ and the gradient is $(2,1)$, where:

• $H$: the Hessian with diagonal entries equal to the curvatures $a_1 = 2$ and $a_2 = 1$
• $H^{-1}$: the inverse of the Hessian, here $\mathrm{diag}(1/2, 1)$
• $\nabla J(\theta_0)$: the gradient at the starting point, here $(2, 1)$
• $d$: the Newton step

Adding gives $\theta_1 = (1,1) + (-1,-1) = (0,0)$, the exact minimum, in a single step. Let us confirm these hand calculations with code so nothing rests on arithmetic we did in our heads.

The run with $\eta = 1.25$ diverges because of the steep direction. On a positive-definite quadratic, an unbounded parameter eventually forces an unbounded objective. Over a finite budget, a lightly weighted coordinate can still look small in the objective while its parameter error matters. Inspect the parameter trajectory alongside the loss when that distinction is relevant.

We now implement least-squares fitting with four synthetic features and an explicit intercept. The design matrix contains a leading column of ones, so the parameter vector has five entries: the intercept followed by four slopes. Including that intercept is essential to this comparison: centering the features can then be absorbed into the intercept without changing the family of predictions. We compare gradient descent with NumPy's least-squares implementation, using the same design matrix and objective.

With an explicit intercept, standardization preserves the set of possible linear predictions. To map standardized slopes back, divide each by its feature standard deviation; subtract the mean-weighted slopes from the standardized intercept. The assertions above verify that this mapping preserves predictions and that the two reference fits attain the same least-squares optimum. The finite-budget badly scaled gradient-descent run can still be farther from that optimum. All 200 rows here are a synthetic training set; in an evaluated model, estimate scaling statistics on its training split and reuse them unchanged for validation and test data.

For this full-rank quadratic, the Hessian spectrum gives the stable interval $0<\eta<2/\lambda_{\max}$. With our half-mean-squared-error objective the Hessian is $X^{\top}X/n$, where $X$ is the $n\times p$ design and $\lambda_{\max}$ is its largest eigenvalue. The next cell computes the eigenvalues and prints $\eta=2/(\lambda_{\min}+\lambda_{\max})$, where $\lambda_{\min}$ is the smallest eigenvalue. This rate minimizes the worst-case Euclidean parameter-error contraction for a fixed step on this quadratic.

The update budget trades computation against closeness to the optimizer. Stopping early can also change a model's generalization behavior; its relation to regularization depends on the model and update rule. Tolerances need to match the units and scale of the parameterization and objective. A small absolute gradient threshold is an algorithmic condition, not a universal accuracy measure.

The next implementation reshuffles the training rows each epoch and updates using the mean gradient of the current batch. We use a nonempty design, nonnegative integer epoch budgets, positive integer batch sizes, a positive initial rate, and nonnegative decay. We count per-example gradient contributions used for updates, and separately note that objective evaluation and diagnostic checks also cost work. A final short mini-batch is divided by its actual number of rows.

On the full-rank least-squares design, the Hessian $X^{\top}X/n$ is constant and positive definite. We can therefore demonstrate the full Newton step directly. The implementation below is specific to this quadratic and does not add a line search or a logistic-loss example. General nonquadratic objectives need the safeguards discussed above.

The printed gradient norm and reference comparison quantify the numerical error. One update reaches the least-squares solution to the reported precision on this design. The exact-arithmetic result is independent of the condition number, while the accuracy of a floating-point linear solve can still depend on scaling and conditioning.

Below are the main parameters that affect how gradient-based optimization behaves, with guidance on how to select them.

• eta scales the negative gradient. For the positive-definite quadratics here, convergence from every start requires $0<\eta<2/\lambda_{\max}$, and $2/(\lambda_{\min}+\lambda_{\max})$ minimizes the worst-case fixed-rate parameter contraction. General objectives need an appropriate step-selection rule.
• steps caps the update count. More iterations can reduce optimization error in a stable run; they do not repair an unstable rate. Early stopping can regularize some fitting procedures, but the optimization examples here do not evaluate that effect on held-out data.
• tol is the absolute gradient-norm threshold used by this illustrative implementation. Interpret it in the current objective and parameter units, and report the returned residual and budget.
• batch_size is the number of examples used per update. It changes gradient noise, work per update, and updates per epoch; 32 is the setting demonstrated here.
• decay controls the displayed schedule $\eta=\eta_0/(1+c\,m/n)$, where $m$ counts per-example gradient contributions already used. The code uses this example-count clock, which differs from counting parameter updates.

Before increasing the budget or loosening a tolerance, check the gradient calculation and the response to the learning rate. A decay schedule and a batch size should be assessed together under a stated budget and sampling rule.

Gradient descent is useful when a differentiable objective is available and first-order updates suit the data and parameter dimensions. A dense linear-model gradient can be cheaper to compute and store than its Hessian. A smooth penalty can be differentiated along with the loss; a nonsmooth penalty or a constraint needs an update rule that handles that additional structure.

Curvature-aware methods are also worth considering when their matrix operations are affordable. Dense Newton steps and quasi-Newton approximations can reduce the number of updates on some problems, but iteration count alone does not measure total cost. Their behavior depends on the objective, derivatives, safeguards and starting point.

On a nonconvex problem, report the objective, gradient norm, stopping reason and initialization. These diagnostics help describe the returned point, but they do not generally prove that it is a local or global minimum. A budget-limited or divergent run must be distinguished from a small-residual termination.

For the positive-definite quadratic examples, the spectrum gives an explicit stable interval and the optimal worst-case fixed rate. For a general objective, local curvature may change as the parameters move. A line search or another step-selection rule can assess proposed displacements; a short successful trial alone does not prove that one fixed rate will remain suitable throughout the run.

When gradient noise persists near an optimum, smaller or decaying steps can reduce fluctuations. Constant steps can still be appropriate, and some problems have vanishing gradient noise at the solution. Inspect trajectories over a stated budget and consider repeated sampling seeds; a flat noisy curve alone is not a convergence proof.

Check the analytic gradient at several representative parameter values against a numerical approximation. A discrepancy can expose a sign, normalization or indexing error, although a finite set of checks is not a proof. During fitting, record enough information to distinguish small gradients, tiny steps, numerical failure and an exhausted budget.

For unpenalized linear least squares with an intercept, invertible feature scaling and centering preserve the prediction family after the corresponding coefficient transformation. They can improve conditioning when unequal feature scales are the cause. They do not remove exact dependence between columns, and equal variances can coexist with strong correlations and a large condition number. A suitably chosen gradient-descent step can converge without standardization.

Estimate preprocessing statistics on the training portion used for fitting and apply the fitted transformation to held-out data. If a penalty is defined in coefficient units, decide how it should transform when those units change: applying the same coefficient penalty after rescaling generally changes the fitting objective. Inspect unusual values and feature scales in the context of the data rather than treating scaling as a substitute for model checks.

A full gradient can be accumulated across smaller data blocks before taking one update, so full-batch descent does not require the entire dataset to remain in memory. Stochastic methods update from individual examples or subsets. Fresh independent sampling and random reshuffling are different procedures; the fixed-parameter unbiasedness result does not by itself cover the adaptive gradients produced during a reshuffled epoch.

A growing or oscillating objective on a fixed quadratic can indicate an unsuitable step. Other causes include derivative mistakes and nonfinite arithmetic. The examples show an explicit instability ceiling, but observations on a general objective need to be interpreted with its changing curvature and update rule.

A sign reversal changes a descent update into an ascent update whenever the gradient is nonzero. Omitting a positive scalar factor can instead act like changing the learning rate; other implementation errors can change the direction or even make the array operations invalid. Numerical checks should test the actual derivative and normalization used by the loop.

Stagnation can arise from a tiny step, finite precision, poor conditioning, or approach to a stationary point. A small parameter change and a small objective change do not distinguish these cases on their own. The gradient residual and stated assumptions provide additional information; even all three diagnostics together do not generally certify a global optimum.

For the dense linear model, a full-gradient update uses $O(np)$ work, a single-example update $O(p)$, and a mini-batch update $O(bp)$. These costs exclude full-objective monitoring and depend on the model's per-example derivative cost. Independent batch averaging reduces covariance by $1/b$ at fixed parameters; the statistical and hardware trade-offs should be assessed for the actual sampling scheme and implementation.

The examples store the full design for simplicity. A first-order implementation can accumulate a full gradient or process updates from a batch while retaining the parameter vector. A dense least-squares Hessian occupies $O(p^2)$ storage, takes $O(np^2)$ work to form directly, and typically takes $O(p^3)$ work to factor; these are costs of the dense formulation rather than all Newton-type methods.

A solver such as scipy.optimize.minimize exposes the BFGS quasi-Newton method for smooth objectives. Compare such a solver with first-order alternatives using the achieved residual, objective, work and memory requirements, rather than assuming a dimension cutoff gives one method a universal advantage. For this chapter's least-squares example, np.linalg.lstsq provides the reference fit.

The training objective measures fitting on the training data. A lower value does not guarantee better held-out performance. Keep optimization diagnostics separate from an evaluation of the fitted model on data and a metric chosen for the application.

Report the criterion that stopped the run, its tolerances, the final objective and gradient norm, and the available budget. Relative objective and parameter changes can supplement the gradient residual. If termination occurred because improvement was small, say so; do not silently treat that condition as proof that the minimizer was reached.

Repeated seeds can show how much a stochastic run varies under its sampling procedure. Logs can also reveal changes in the observed trajectory, but they do not identify a unique cause such as data drift or an incorrect step size. Investigate the data, derivatives and update rule together when behavior changes.

Gradient descent uses the gradient to construct a local update. On a positive-definite quadratic, eigenvalue analysis determines the stable interval and the best worst-case fixed-rate contraction. The scalar recurrence $\theta_k=(1-\eta a)^k\theta_0$ makes convergence, sign alternation and divergence explicit. These exact statements require the stated quadratic conditions; a general nonconvex objective needs additional analysis.

The examples also show why the comparison setup matters. One common absolute rate gave the two quadratics identical shallow-coordinate progress; applying the same reciprocal-curvature rate rule separately exposed a much faster well-conditioned run. In least squares, keeping an explicit intercept made standardized and unscaled reference fits equivalent, while their finite-budget gradient-descent progress differed. Newton reached the quadratic solution to the reported numerical precision with one update.

Batch and stochastic variants use different amounts of data per update and require explicit sampling and work-count conventions. A numerical gradient check can catch implementation mistakes, and stopping diagnostics describe what a run achieved. Curvature, feature coordinates, numerical arithmetic and gradient noise all belong in that interpretation; a termination message alone is not a certificate of global optimality.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about gradient descent, curvature, learning-rate stability, and numerical optimization checks.