Support Vector Machines and Support Vector Regression › Implementation in Scikit-learn

We build the linear hinge-loss classifier from primitives first, then compare it with scikit-learn's SVC on the same data. Working from first principles helps us see where the subgradient comes from and how sensitive the solution is to feature scaling. We standardize the numerical inputs here so that both optimizers use the same feature representation.

We reuse the toy two-dimensional dataset from the visualization section. The full toy set acts as training data here; this example has no held-out performance estimate. We compute the means and standard deviations from those training inputs only, then supply the same standardized features to both optimizers. Scaling changes the geometry of the weight penalty and kernel distances. A feature with large numerical units can contribute heavily to a distance calculation, but that does not imply that an SVM must ignore every smaller-scale feature.

This fixed-step subgradient run uses a budget of 3,000 updates; that budget does not certify convergence to the optimum. We print the training accuracy, the weight norm $\|\mathbf{w}\|$, and the count of hinge-active examples. The count records strict margin violations, $m_i<1$, at the current iterate. It includes misclassified points as well as correctly classified points too close to the boundary.

Now compare with the reference SVC, using the same $C$ and standardized inputs. Its printed margin width is measured in the standardized feature coordinates supplied to this fit. The reference solver works with the dual quadratic program, while our scratch implementation descends on the primal hinge objective. They target the same normalized hinge objective, but matching training labels alone does not establish that their parameter estimates or objective values agree.

The printed label agreement and objective gap answer different questions. Matching labels on this easy training set can coexist with a noticeable objective gap, because a fixed step size can keep the scratch iterate moving around the nonsmooth optimum. The scratch objective uses $\tfrac{1}{2}\|\mathbf{w}\|^2 + C\sum_i \xi_i$, matching the reference SVC normalization. The scratch hinge-active fraction counts strict margin violations at its current iterate; it is not the dual support-vector fraction. The reference support count comes separately from svc_ref.support_.

The SVR side follows the same pattern. We reuse the noisy sine dataset from the visualization and inspect epsilon-insensitive loss and tube occupancy alongside the reference support-vector set. Points strictly inside the tube have zero epsilon-insensitive loss and zero dual coefficients. Points on the tube edges can have nonzero coefficients, as can points outside; the support set is not limited to strict outside points.

The printed tube occupancy directly counts residuals with $|r| \leq \varepsilon$, using the fitted predictions. The mean epsilon-insensitive loss and mean squared residual summarize different loss functions and scale differently when target units change; their relative magnitudes do not reveal how many observations lie inside the tube. The reference support set can include points on the tube boundary, and finite solver tolerance can put their computed residuals slightly inside or outside it. These in-sample diagnostics describe the fit; they do not establish that the model generalizes or that a particular tube width is optimal.

The tube width is a tuning choice, not a fixed property of the data. Refitting SVR on the same sine training set over a small epsilon grid shows how the support fraction and the fraction of in-sample residuals inside the tube vary. Here the support fraction decreases across the grid, while the measured inside-tube fraction rises overall and is unchanged between epsilon 0.2 and 0.4. These counts use the fitted residuals and the exact comparison shown in the code, so solver tolerance can affect points near the tube edges. This is a training-set diagnostic, not a validation comparison.