Linear Classifiers › One-vs-Rest Multiclass Strategy

Before softmax became standard, and still useful in some settings, the one-vs-rest (OvR) strategy handles multiclass problems by training $K$ separate binary classifiers. Classifier $k$ is trained to distinguish class $k$ from all other classes. At prediction time, all $K$ classifiers produce a score, and the class with the highest score is predicted.

OvR is simple to implement: you just train $K$ logistic regression models independently, one for each class. Each classifier sees the full training set, with the positive class being the $k$-th class and the negative class being all other examples combined. This works well when the classes are well-separated and the models' output scores are on comparable scales.

The disadvantage of OvR is that the $K$ classifiers are not jointly trained to produce a coherent probability distribution. The scores from different classifiers are not comparable, and they do not necessarily sum to 1. Softmax (multinomial logistic regression) avoids this by training all $K$ weight vectors jointly and normalizing the outputs. In practice, multinomial logistic regression with softmax outperforms OvR, especially when classes are similar and the probability estimates need to be well-calibrated. Scikit-learn's multi_class='multinomial' parameter enables this.