Support Vector Machines and Support Vector Regression › Summary

Support Vector Machines connect a geometric margin objective to convex optimization. For separable data, the hard-margin problem requires every functional margin to be at least one. The soft-margin problem relaxes that requirement using slack penalties. Eliminating those slacks gives the hinge-loss objective $\tfrac12\|\mathbf{w}\|^2+C\sum_i\max(0,1-y_i f(\mathbf{x}_i))$. This finite penalty allows violations rather than preserving the hard constraint. Dividing by $nC$ gives a mean-loss objective with penalty $\lambda\|\mathbf{w}\|^2/2$, where $\lambda=1/(nC)$. Under that convention, larger $C$ means weaker relative regularization.

The Lagrangian view exposes a dual problem that depends on pairwise inner products. Replacing them by a valid PSD kernel gives a feature-space inner product and preserves concavity of the dual objective. Linear, appropriately parameterized polynomial, and RBF kernels are examples. Arbitrary similarities need not be valid, and checking one Gram matrix is not a proof for every possible input set. KKT complementarity sets the multiplier to zero for a point strictly beyond its margin hyperplane; the remaining support-vector terms determine the decision sum, although the support set may include all training examples.

Support Vector Regression uses an $\varepsilon$-insensitive loss that is zero for $|r|\le\varepsilon$ and linear beyond the tube. Support vectors may lie on the tube edges as well as outside it, and their number is data dependent. This loss limits the influence of large response residuals relative to squared loss; it does not ensure robustness to high-leverage inputs or improved validation error. Choose suitable preprocessing, validate the relevant penalty and kernel settings, account for class imbalance when it matters, and assess calibration when probabilities are required. The examples also show a limit of the geometric intuition: a soft-margin optimum can be a constant classifier with $\mathbf{w}=0$.