Support Vector Machines and Support Vector Regression › Visualizing Support Vector Machines

The following cell generates a small two-dimensional toy dataset and fits a large-$C$ linear SVM, producing the decision-score field used by the margin figure below. A large penalty can recover a separating solution on this example; it is still a soft-margin SVC fit. The reference estimator uses labels 0 and 1; our signed mathematical labels are $-1$ and $+1$. We map 0 to $-1$ and 1 to $+1$ when calculating functional margins.

The dual view gives a second way to read the fitted margin. In the large-C fit above, most training points have zero dual coefficient, while the support vectors carry positive coefficient magnitudes. The next cell records those coefficients and the functional margins used by the figure.

The hinge loss makes the margin behavior quantitative. The next figure compares the hinge, logistic, and zero-one losses as functions of the functional margin $m = y\,f(\mathbf{x})$, where $y \in \{-1, +1\}$ and $f(\mathbf{x}) = \mathbf{w}^\top \mathbf{x} + b$. All three approach zero for very confident correct predictions, but the hinge loss transitions sharply at $m = 1$; its flat region for $m > 1$ removes those points from the hinge subgradient. At an optimum, KKT complementarity also forces their dual coefficients to zero. The resulting support set can be sparse; its size depends on the data and fit.

For the plotted zero-one margin loss, a zero-margin tie counts as an error. The scratch prediction code later chooses class 0 at an exactly zero score; tie handling is a stated convention rather than a property shared by every classifier API.

The following cell computes the loss curves that the figure below renders.

The loss curves are computed in the preceding cell and only rendered here.

The kernel trick changes nonlinear behavior substantially. The next figures compare linear and RBF SVMs on data with a curved decision boundary and independently seeded label flips at an 8% probability. Both models use C=3, so the penalty is held constant.

The training diagnostics are computed on the same 120 observations with the same penalty C. They show how each hypothesis class fits this noisy example; they do not estimate performance on new data. The RBF fit has higher training accuracy here, and its plot shows a nonlinear boundary around the main radial structure. The linear model predicts one class across the displayed window. The support counts describe the fitted representations; they do not independently establish which model generalizes better. The independent seed for label flips leaves the earlier data generator's stream unchanged for the later SVR example.

The $\varepsilon$-insensitive loss is zero for $|r| \leq \varepsilon$ and grows linearly outside that interval. The linear tails reduce the influence of large response residuals relative to squared loss, without guaranteeing resistance to high-leverage inputs. The following cell computes the three loss curves used in the figure below.

An SVR fit shows the tube explicitly. The band of width $2\varepsilon$ surrounds the model prediction, and training points inside it incur zero data-fit loss. The weight penalty remains part of the full objective. The following cell fits the SVR and computes the prediction curve used in the figure below.