Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Differential Calculus and Optimization Basics › Unconstrained Optimization

Michael BrenndoerferOctober 24, 202517 min read

Quantitative FinanceAll chapters

Part 1 of 15Mathematical Foundations

Chapter 6Differential Calculus and Optimization Basics › Unconstrained Optimization

Reading section 4 of 11 · Unconstrained Optimization · 17 minSection 4 of 11 · 17 min

Section 4 of 11 · 17 min read

Differential Calculus and Optimization Basics › Unconstrained OptimizationOne section · Sections in this chapter4 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Optimization seeks the best objective value among feasible alternatives, when an optimum exists. In unconstrained optimization, we minimize or maximize without explicit constraints beyond the function's domain.

"Best" might mean smallest for costs and risks, or largest for returns and profits. The quadratic examples here are defined on all of $\mathbb{R}^2$ , so their inputs have no domain boundaries or additional restrictions. Other unconstrained objectives can still have an intrinsic domain that an algorithm must respect. The stationarity principles developed here also help us handle explicit constraints later.

First-Order ConditionsLink Copied

For a differentiable unconstrained problem, we begin by identifying interior stationary points, where the gradient equals zero. In this smooth setting we also call them critical points; they are candidates, not guaranteed optima.

The first-order condition follows from the gradient's directional interpretation. At a differentiable interior local minimum, there is no improving decrease in a sufficiently small neighborhood; at a local maximum, there is no improving increase. A nonzero gradient would give local ascent along the gradient and local descent along its negative. The appropriate one of these directions contradicts either type of local extremum, so its gradient must be zero. Higher-order increases or decreases can still occur when first-order rates vanish.

First-Order Necessary Condition

If $f$ has a local minimum or maximum at an interior point $x^*$ , and $f$ is differentiable at $x^*$ , then:

\nabla f(x^*) = 0

Points satisfying this condition are called critical points or stationary points.

The first-order condition is necessary but not sufficient for an optimum. A zero gradient tells us we have found a stationary point, but not all stationary points are minima or maxima. Some are saddles, where nearby values increase in some directions and decrease in others. An idealized mountain pass is a local minimum along the ridge between peaks and a local maximum along a route crossing between valleys. Both first-order directional rates vanish at the pass, yet higher-order changes have opposite signs. Degenerate stationary points can need further analysis too.

Setting the gradient to zero gives us a system of equations. For our earlier quadratic function:

\nabla f = \begin{pmatrix} 2x - 2y + 4 \\ 4y - 2x - 6 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}

Solving this system:

In[13]:

Code

from scipy.optimize import fsolve


def gradient_system(vars):
    x, y = vars
    return [2 * x - 2 * y + 4, 4 * y - 2 * x - 6]


critical_point, root_info, root_status, root_message = fsolve(
    gradient_system, [0, 0], full_output=True
)
if (
    root_status != 1
    or not np.all(np.isfinite(critical_point))
    or not np.all(np.isfinite(root_info["fvec"]))
    or np.max(np.abs(root_info["fvec"])) > 1e-10
):
    raise RuntimeError(f"Stationary-point solve failed: {root_message}")

Out[14]:

Console

Critical point: x = -1.0000, y = 1.0000
Function value at critical point: -5.0000
Gradient at critical point: [0. 0.]

The critical point is at (-1, 1), where the gradient is indeed zero.

Second-Order Conditions and the HessianLink Copied

A zero gradient does not classify a stationary point. Second-order information in the Hessian provides decisive tests in the definite and indefinite cases below; some degenerate cases need higher-order or direct analysis.

The Hessian matrix extends the concept of the second derivative to multiple dimensions. Just as the second derivative of a single-variable function tells us about the function's curvature (whether it curves upward or downward), the Hessian tells us about curvature in all directions simultaneously. Because there are infinitely many directions in multidimensional space, this information is naturally organized into a matrix rather than a single number.

Hessian Matrix

The Hessian matrix of a function $f: \mathbb{R}^n \to \mathbb{R}$ is the $n \times n$ matrix of second partial derivatives:

H = \begin{pmatrix} \frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \cdots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\ \frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \cdots & \frac{\partial^2 f}{\partial x_2 \partial x_n} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial^2 f}{\partial x_n \partial x_1} & \frac{\partial^2 f}{\partial x_n \partial x_2} & \cdots & \frac{\partial^2 f}{\partial x_n^2} \end{pmatrix}

For functions with continuous second derivatives, the Hessian is symmetric.

The diagonal entries of the Hessian capture the pure second derivatives: how the rate of change with respect to one variable itself changes as that variable varies. The off-diagonal entries capture the mixed partial derivatives: how the rate of change with respect to one variable changes as another variable varies. The symmetry of the Hessian, known as the equality of mixed partials or Clairaut's theorem, holds for functions with continuous second derivatives and reflects the fundamental fact that the order of differentiation doesn't matter for such functions.

At an interior stationary point where a valid second-order expansion exists, a definite or indefinite Hessian gives the classification below. Continuous second derivatives nearby are a sufficient smoothness condition. Mixed positive and negative eigenvalues identify a saddle even if other eigenvalues are zero. If zero eigenvalues occur without mixed signs, the quadratic test is inconclusive and higher-order or other analysis is required.

All eigenvalues positive: strict local minimum
All eigenvalues negative: strict local maximum
Mixed signs: saddle point

The connection between eigenvalues and the nature of critical points comes from a deep result in linear algebra. The eigenvalues of the Hessian represent the curvatures of the function along the principal directions (the eigenvectors). If all these curvatures are positive, the function curves upward in every direction, creating a bowl shape characteristic of a minimum. If all are negative, it curves downward in every direction, creating an inverted bowl characteristic of a maximum. Mixed signs create the saddle shape, curving up in some directions and down in others.

For our quadratic function, the Hessian is constant:

H = \begin{pmatrix} 2 & -2 \\ -2 & 4 \end{pmatrix}

In[15]:

Code

import numpy as np

# Compute Hessian analytically (constant for quadratic)
# For f(x,y) = x² + 2y² - 2xy + 4x - 6y:
# ∂²f/∂x² = 2, ∂²f/∂y² = 4, ∂²f/∂x∂y = -2
H = np.array([[2, -2], [-2, 4]])

# Find eigenvalues to classify the critical point
eigenvalues = np.linalg.eigvals(H)

Out[16]:

Console

Hessian matrix:
[[ 2 -2]
 [-2  4]]

Eigenvalues: [0.76393202 5.23606798]

All eigenvalues positive: True
Classification: Local minimum

Both eigenvalues are positive, confirming that the critical point (-1, 1) is a local minimum. Since the function is quadratic with a positive definite Hessian, this minimum is also global.

The Hessian is constant here because two derivatives of a quadratic term leave a constant. Its directional curvature is therefore the same everywhere, simplifying analysis and computation. For a more general function, evaluate the Hessian at the stationary point; if its test is inconclusive, further analysis is still needed.

Out[17]:

Visualization

Contour plot with elliptical contours centered on the minimum at (-1, 1), marked by a star. — Contour plot of the quadratic function showing the global minimum at (-1, 1). The elliptical contours indicate the function's positive definite Hessian.

Out[18]:

Visualization

3D surface plot of a bowl-shaped quadratic function illustrating a local minimum. — Minimum: all eigenvalues positive.

3D surface plot of an inverted bowl-shaped quadratic function illustrating a local maximum. — Maximum: all eigenvalues negative.

3D surface plot of a saddle-shaped function illustrating a saddle point. — Saddle point: mixed eigenvalues.

Gradient DescentLink Copied

Gradient descent is a first-order optimization algorithm that seeks a minimum by repeatedly stepping opposite the gradient. Convergence requires appropriate objective, initialization, and step-size conditions.

Gradient descent is conceptually straightforward. At any point, we compute the gradient to determine the direction of steepest ascent, then move in the opposite direction to descend toward lower values. By repeating this process, we trace out a path that, under appropriate conditions, leads to a minimum. This iterative approach is particularly valuable for high-dimensional problems where analytical solutions are impossible or impractical.

The update rule is:

x_{k+1} = x_k - \alpha \nabla f(x_k)

where:

$x_k$ : the current iterate at step $k$
$x_{k+1}$ : the next iterate after the update
$\alpha > 0$ : the learning rate (step size), controlling how far we move in each iteration
$\nabla f(x_k)$ : the gradient evaluated at the current point, indicating the direction of steepest ascent

The negative sign chooses a local first-order descent direction when the gradient is nonzero. It does not guarantee that a finite update lowers the objective: the learning rate $\alpha$ scales the displacement, and an excessive step can overshoot.

The choice of $\alpha$ controls whether the algorithm converges. If it is too large, the algorithm may overshoot and diverge. If it is too small, convergence becomes painfully slow.

Increasing the learning rate within a stable range can accelerate progress, while excessive steps can overshoot or diverge. The stable range depends on curvature and other objective properties, not just distance from a minimum. For a smooth convex objective, a sufficiently small step relative to its gradient-Lipschitz bound can give stable descent; a very small step can be slow. Momentum combines current and past gradient information, whereas adaptive learning-rate methods rescale steps using gradient history. Neither mechanism guarantees convergence for every problem.

In[19]:

Code

import numpy as np


def gradient_descent(f, grad_f, x0, learning_rate=0.1, max_iter=200, tol=1e-6):
    """
    Record an educational fixed-step gradient-descent trajectory.

    Parameters:
    f: objective function
    grad_f: gradient function
    x0: starting point
    learning_rate: step size
    max_iter: maximum iterations
    tol: convergence tolerance on gradient norm

    Returns:
    x_history: array of iterates, shape (updates + 1, dimension)
    f_history: array of function values, shape (updates + 1,)
    Histories include the initial point and do not certify convergence.
    """
    if np.iscomplexobj(x0):
        raise ValueError("x0 must be real")
    x = np.asarray(x0, dtype=float).copy()
    if x.ndim != 1 or x.size == 0 or not np.all(np.isfinite(x)):
        raise ValueError("x0 must be a nonempty finite vector")
    if not np.isfinite(learning_rate) or learning_rate <= 0:
        raise ValueError("learning_rate must be finite and positive")
    if not isinstance(max_iter, (int, np.integer)) or max_iter < 0:
        raise ValueError("max_iter must be a nonnegative integer")
    if not np.isfinite(tol) or tol < 0:
        raise ValueError("tol must be finite and nonnegative")

    def objective_at(point):
        value = f(point)
        if np.ndim(value) != 0 or np.iscomplexobj(value) or not np.isfinite(value):
            raise ValueError("f must return a finite real scalar")
        return float(value)

    x_history = [x.copy()]
    f_history = [objective_at(x)]

    for i in range(max_iter):
        raw_grad = grad_f(x)
        if np.iscomplexobj(raw_grad):
            raise ValueError("grad_f must return a real vector")
        grad = np.asarray(raw_grad, dtype=float)
        if grad.shape != x.shape or not np.all(np.isfinite(grad)):
            raise ValueError("grad_f must return a finite vector matching x")

        # Scale first so squaring very large or tiny components does not
        # overflow or underflow in the stopping calculation.
        grad_scale = np.max(np.abs(grad))
        grad_norm = (
            0.0 if grad_scale == 0 else grad_scale * np.linalg.norm(grad / grad_scale)
        )
        if not np.isfinite(grad_norm):
            raise ValueError("Gradient norm is not representable")
        if grad_norm < tol:
            break

        # Update step
        x = x - learning_rate * grad
        if not np.all(np.isfinite(x)):
            raise ValueError("Gradient step produced a nonfinite iterate")
        x_history.append(x.copy())
        f_history.append(objective_at(x))

    return np.array(x_history), np.array(f_history)


# Run gradient descent on our quadratic function


def f_vec(x):
    return x[0] ** 2 + 2 * x[1] ** 2 - 2 * x[0] * x[1] + 4 * x[0] - 6 * x[1]


def grad_f_vec(x):
    return np.array([2 * x[0] - 2 * x[1] + 4, 4 * x[1] - 2 * x[0] - 6])


x0 = np.array([3.0, 0.0])
x_history, f_history = gradient_descent(f_vec, grad_f_vec, x0, learning_rate=0.15)

Out[20]:

Console

Starting point: [3. 0.]
Final point: [-0.999999, 1.000001]
True minimum: [-1, 1]
Updates completed: 120
Final gradient norm: 9.98e-07
Gradient tolerance met: True
Final function value: -5.000000

Out[21]:

Visualization

Contour plot showing gradient descent iterates moving from (3, 0) toward the minimum at (-1, 1). — Gradient descent path from (3, 0) toward the analytic minimum at (-1, 1); the displayed endpoint meets the stated gradient tolerance.

Log-scale plot of f(x_k) minus the optimum value versus iteration, showing convergence. — Quadratic optimality gap evaluated from displacement to the known optimum; values below 10^-16 would be displayed at that floor, not certified to that accuracy.

For this smooth strongly convex quadratic, the Hessian's largest eigenvalue is about 5.236, so the fixed step $\alpha=0.15$ lies within the stable interval $0<\alpha<2/L$ . With the 200-iteration cap, the run satisfies the gradient-norm tolerance $10^{-6}$ before stopping and exhibits linear convergence. Convexity alone does not make an arbitrary step size converge.

Out[22]:

Visualization

Contour plot with gradient descent path for learning rate 0.01 showing slow progress toward the minimum. — Learning rate 0.01: slow progress over 50 updates; the star is the analytic reference minimum.

Contour plot with gradient descent path for learning rate 0.15 showing steady convergence to the minimum. — Learning rate 0.15: faster progress over the same 50-update budget, not a tolerance-certified final point.

Contour plot with gradient descent path for learning rate 0.35 showing oscillatory convergence toward the minimum. — Learning rate 0.35: stable oscillatory progress over 50 updates; the common limits include its early excursion.