Support Vector Machines and Support Vector Regression

A classifier can fit every training label correctly and still fail to minimize the SVM objective. That happens in the toy example below: our fixed-step implementation agrees with the reference SVC on all training labels, while the reference fit has a lower value of the same hinge-loss objective. The scratch algorithm uses a fixed learning rate and a finite update budget without a convergence certificate. The Support Vector Machine (SVM) gives us an objective for assessing that optimization gap. For linearly separable data, its hard-margin form chooses the separating hyperplane with the widest buffer to the nearest training points. Its soft-margin form allows violations of that buffer in exchange for a penalty. We will derive both formulations and use the dual coefficients to identify which training points contribute to the fitted decision function.

The dual formulation gives a precise notion of sparsity. In a soft-margin SVM, training examples with zero dual coefficient do not contribute to the decision function. Only points on or inside the margin can have nonzero coefficients, although some such coefficients may also be zero. This differs from Lasso's sparsity in input-feature coefficients, and it can be useful computationally when the support set is small. A single formula, hinge loss plus an L2 penalty, connects the geometric story of maximum margins to regularized empirical risk minimization.

The same machinery extends to regression. Support Vector Regression (SVR) uses an $\varepsilon$-insensitive loss. Residuals with magnitude at most $\varepsilon$ incur no data-fit loss, while excess residual magnitude is penalized linearly. Points strictly inside the tube have zero dual coefficients, which can produce a sparse model. Compared with squared loss, linear growth limits the influence of large response residuals, although it does not protect against every outlier or guarantee a small support set.

We work through margins, support vectors, the hinge loss, and the soft-margin Lagrangian, then connect valid kernels and ε-insensitive regression to executable examples. The margin and dual derivations spell out the Lagrangian steps.