Support Vector Machines and Support Vector Regression › Advantages

The maximum-margin principle gives SVMs a clean, geometric objective. With a valid positive-semidefinite kernel, the standard SVM objective is convex: every local minimum is global. Suitable solvers can approach a global optimum, but their stopping tolerances, update rules, and convergence conditions still matter. Support vectors identify the examples that contribute to the decision function, which can make the fit compact and useful to inspect. The support set can also contain most or all of the training data, and convexity alone does not make every fitted parameter unique.

The kernel trick extends the same dual optimization framework to nonlinear boundaries by replacing inner products with a valid kernel. A real-valued kernel must be symmetric and produce a positive-semidefinite Gram matrix for every finite collection of inputs; it then corresponds to inner products in a feature space. Support Vector Regression uses the same representation with an $\varepsilon$-insensitive loss. That loss ignores residuals within the tube and grows linearly outside it, reducing the influence of large response residuals relative to squared loss. Its support set need not be small, and whether it predicts better than OLS depends on the data and must be checked on validation data.