Gradient Descent and Numerical Optimization › Practical Implications

Gradient descent is useful when a differentiable objective is available and first-order updates suit the data and parameter dimensions. A dense linear-model gradient can be cheaper to compute and store than its Hessian. A smooth penalty can be differentiated along with the loss; a nonsmooth penalty or a constraint needs an update rule that handles that additional structure.

Curvature-aware methods are also worth considering when their matrix operations are affordable. Dense Newton steps and quasi-Newton approximations can reduce the number of updates on some problems, but iteration count alone does not measure total cost. Their behavior depends on the objective, derivatives, safeguards and starting point.

On a nonconvex problem, report the objective, gradient norm, stopping reason and initialization. These diagnostics help describe the returned point, but they do not generally prove that it is a local or global minimum. A budget-limited or divergent run must be distinguished from a small-residual termination.

For the positive-definite quadratic examples, the spectrum gives an explicit stable interval and the optimal worst-case fixed rate. For a general objective, local curvature may change as the parameters move. A line search or another step-selection rule can assess proposed displacements; a short successful trial alone does not prove that one fixed rate will remain suitable throughout the run.

When gradient noise persists near an optimum, smaller or decaying steps can reduce fluctuations. Constant steps can still be appropriate, and some problems have vanishing gradient noise at the solution. Inspect trajectories over a stated budget and consider repeated sampling seeds; a flat noisy curve alone is not a convergence proof.

Check the analytic gradient at several representative parameter values against a numerical approximation. A discrepancy can expose a sign, normalization or indexing error, although a finite set of checks is not a proof. During fitting, record enough information to distinguish small gradients, tiny steps, numerical failure and an exhausted budget.

For unpenalized linear least squares with an intercept, invertible feature scaling and centering preserve the prediction family after the corresponding coefficient transformation. They can improve conditioning when unequal feature scales are the cause. They do not remove exact dependence between columns, and equal variances can coexist with strong correlations and a large condition number. A suitably chosen gradient-descent step can converge without standardization.

Estimate preprocessing statistics on the training portion used for fitting and apply the fitted transformation to held-out data. If a penalty is defined in coefficient units, decide how it should transform when those units change: applying the same coefficient penalty after rescaling generally changes the fitting objective. Inspect unusual values and feature scales in the context of the data rather than treating scaling as a substitute for model checks.

A full gradient can be accumulated across smaller data blocks before taking one update, so full-batch descent does not require the entire dataset to remain in memory. Stochastic methods update from individual examples or subsets. Fresh independent sampling and random reshuffling are different procedures; the fixed-parameter unbiasedness result does not by itself cover the adaptive gradients produced during a reshuffled epoch.

A growing or oscillating objective on a fixed quadratic can indicate an unsuitable step. Other causes include derivative mistakes and nonfinite arithmetic. The examples show an explicit instability ceiling, but observations on a general objective need to be interpreted with its changing curvature and update rule.

A sign reversal changes a descent update into an ascent update whenever the gradient is nonzero. Omitting a positive scalar factor can instead act like changing the learning rate; other implementation errors can change the direction or even make the array operations invalid. Numerical checks should test the actual derivative and normalization used by the loop.

Stagnation can arise from a tiny step, finite precision, poor conditioning, or approach to a stationary point. A small parameter change and a small objective change do not distinguish these cases on their own. The gradient residual and stated assumptions provide additional information; even all three diagnostics together do not generally certify a global optimum.

For the dense linear model, a full-gradient update uses $O(np)$ work, a single-example update $O(p)$, and a mini-batch update $O(bp)$. These costs exclude full-objective monitoring and depend on the model's per-example derivative cost. Independent batch averaging reduces covariance by $1/b$ at fixed parameters; the statistical and hardware trade-offs should be assessed for the actual sampling scheme and implementation.

The examples store the full design for simplicity. A first-order implementation can accumulate a full gradient or process updates from a batch while retaining the parameter vector. A dense least-squares Hessian occupies $O(p^2)$ storage, takes $O(np^2)$ work to form directly, and typically takes $O(p^3)$ work to factor; these are costs of the dense formulation rather than all Newton-type methods.

A solver such as scipy.optimize.minimize exposes the BFGS quasi-Newton method for smooth objectives. Compare such a solver with first-order alternatives using the achieved residual, objective, work and memory requirements, rather than assuming a dimension cutoff gives one method a universal advantage. For this chapter's least-squares example, np.linalg.lstsq provides the reference fit.

The training objective measures fitting on the training data. A lower value does not guarantee better held-out performance. Keep optimization diagnostics separate from an evaluation of the fitted model on data and a metric chosen for the application.

Report the criterion that stopped the run, its tolerances, the final objective and gradient norm, and the available budget. Relative objective and parameter changes can supplement the gradient residual. If termination occurred because improvement was small, say so; do not silently treat that condition as proof that the minimizer was reached.

Repeated seeds can show how much a stochastic run varies under its sampling procedure. Logs can also reveal changes in the observed trajectory, but they do not identify a unique cause such as data drift or an incorrect step size. Investigate the data, derivatives and update rule together when behavior changes.