Calibration and Parameter Estimation in Models › Optimization Techniques

Once we've specified an objective function, we need algorithms to find the minimizing parameters. Calibration optimization presents particular challenges: objective functions may be non-convex with multiple local minima, parameter spaces may be constrained, and model evaluations (especially for derivatives requiring Monte Carlo or PDE methods) may be computationally expensive. Choosing an optimization technique requires matching these constraints to the algorithm's capabilities.

When the objective function is smooth and its gradient is available (analytically or via automatic differentiation), gradient-based methods can be efficient. The simplest is gradient descent, which takes steps opposite the gradient:

$$\theta_{k+1} = \theta_k - \alpha_k \nabla L(\theta_k)$$

where:

• $\theta_k$: parameter vector at iteration $k$
• $\alpha_k$: step size (learning rate) controlling how far we move at each iteration
• $\nabla L(\theta_k)$: gradient of the objective function, pointing in the direction of steepest increase

For a differentiable objective and a nonzero gradient, moving opposite the gradient gives a local descent direction in the usual Euclidean coordinates. A sufficiently small positive step reduces the loss, provided the step remains in the objective's domain and respects any parameter constraints. An arbitrary finite step need not reduce the loss: a step that is too large can overshoot and increase it. At a zero gradient, this update leaves the parameters unchanged and does not by itself establish a minimum.

BFGS and L-BFGS build approximate curvature information from changes in gradients. Under suitable regularity and line-search conditions near an optimum, they can converge faster than basic gradient descent. This is a conditional benefit, not a guarantee for every calibration objective.

Gradient methods can find local minima efficiently under suitable conditions, but non-convex objectives can also contain stationary points that are not minima and suboptimal local minima. Initialization and local curvature affect the outcome. This limitation motivates the global optimization methods discussed next.

For non-convex objectives, stochastic global-search methods seek broader exploration through populations or moves between regions. A finite run does not guarantee thorough coverage or a global optimum:

Differential evolution maintains a population of candidates and proposes new vectors through mutation and crossover, retaining candidates according to their objective values. It is derivative-free. Bounds and general constraints require explicit handling by the selected implementation; constraint satisfaction is not intrinsic to the mutation formula.

Simulated annealing can accept uphill moves with probability that decreases as the temperature falls, for a fixed loss increase. Cooling concentrates the search, but convergence claims require assumptions about the schedule and proposals. The analogy is annealing in solids, where controlled heating and cooling can allow lower-energy configurations; it is not a universal material-strengthening rule.

Basin hopping perturbs a candidate, locally minimizes from the perturbed point, and accepts or rejects the resulting basin, while retaining the lowest minimum found. It can revisit the same basin. Independent additional runs can help assess sensitivity to initialization without proving global optimality.

These methods often explore more broadly than one local run, usually at greater function-evaluation cost; sensitivity to initialization remains problem dependent. A hybrid can use a broad search to find a promising region and then refine it with a local method.

Model parameters often have economic constraints that must be respected. Volatility is nonnegative in general and must be strictly positive where the chosen formula or parameterization requires it. Mean-reversion speeds should be positive for stability. Correlation parameters must lie in $[-1, 1]$. These constraints can be enforced through several mechanisms:

Constrained optimization: SLSQP and constrained trust-region methods accept supplied bounds and general constraints and seek feasible solutions to a specified tolerance. This does not guarantee that every intermediate objective evaluation is feasible. Model evaluations must handle possible trial points, and final constraint residuals must be checked.

Parameter transformation: For a strictly positive parameter $\sigma$, finite real coordinates $\eta$ map through $\sigma=\exp(\eta)$ in exact arithmetic. For a correlation, $\rho=\tanh(\tilde{\rho})$ covers $(-1,1)$ at finite coordinates, excluding the endpoints. These maps remove the represented interior scalar constraints; coupled constraints still require separate handling. Floating-point overflow and underflow also need numerical checks.

Penalty methods: Add a large penalty to the objective when constraints are violated:

$$L_{penalized}(\theta) = L(\theta) + \lambda \sum_j \max(0, g_j(\theta))^2$$

where:

• $L_{penalized}(\theta)$: penalized objective function that combines fit quality and constraint satisfaction
• $L(\theta)$: original loss function measuring fit to data
• $\lambda$: penalty parameter controlling the severity of constraint violation penalties
• $g_j(\theta) \leq 0$: inequality constraints (positive values indicate violation)
• $\theta$: model parameters

For a fixed violation, increasing $\lambda$ increases its cost. A finite quadratic penalty generally encourages approximate rather than exact feasibility, and arbitrarily small violations have arbitrarily small penalties. Report residual constraint violations explicitly and use a suitable hard-constraint method when required.

Regularization adds penalties for large parameter values or parameter instability, encouraging parsimonious solutions:

$$L_{regularized}(\theta) = L(\theta) + \lambda \|\theta\|^2$$

where:

• $L_{regularized}(\theta)$: regularized objective function balancing fit and parameter magnitude
• $L(\theta)$: original loss function
• $\lambda$: regularization strength controlling the trade-off
• $\|\theta\|^2$: squared Euclidean norm of the parameters

This L2 penalty encourages a smaller parameter norm in the chosen coordinates. Its meaning depends on units, scaling and centering; it is not automatically a penalty on economic complexity or changes over time. A temporal stability penalty would instead compare successive parameter vectors. Regularization can reduce variance while adding bias, so choose its strength using an appropriate validation design rather than assuming it improves generalization.