Gradient Descent and Numerical Optimization › Advantages

The update needs an objective and its gradient rather than a closed-form minimizer. We can therefore use it for smooth fitting problems such as logistic regression, using negative log-likelihood as the objective. For an unconstrained problem, the same update structure applies when we change the loss and compute its new gradient. Nondifferentiable objectives require an appropriate alternative, such as a subgradient or proximal method.

For the dense linear model used here, a full gradient costs $O(np)$ operations for $n$ rows and $p$ parameters. A single-example or mini-batch update can use less data before changing the parameters. These first-order updates avoid forming a $p\times p$ Hessian, and data can be supplied in batches when storing the full design is inconvenient.

An iterative fit also produces a history that we can inspect. We can record the objective, gradient norm, parameter changes, and stopping reason. Computing these diagnostics has a cost, and an increasing objective can be expected during stochastic updates, so the appropriate interpretation depends on the algorithm.

Smooth penalties can be included in the objective and differentiated along with the data-fit term. Constraints need additional treatment, such as a projection or a solver designed for the feasible set; an unconstrained gradient step alone does not enforce them.