Classification Applications › Evaluation and Metrics

Classification evaluation requires care about which metrics you report and what they mean.

Accuracy, the fraction of correct predictions, is the most natural metric but fails in two important ways.

First, it treats all errors equally. Misclassifying a fraud report as an account inquiry has far higher cost than misclassifying an account inquiry as a fraud report. Applications with asymmetric error costs need weighted or asymmetric metrics.

Second, accuracy is misleading for imbalanced datasets. A classifier that always predicts the majority class in a 95/5 imbalance achieves 95% accuracy. That number is meaningless as an evaluation of the classifier's utility.

The combination of these two failures makes accuracy a particularly poor guide for systems where the economically important class is the rare one. In fraud detection, spam filtering, medical diagnosis, and content moderation, the class that matters is almost always the minority class. Reporting accuracy in these settings creates a false sense of security: a model with 98% accuracy can still be missing 40% of fraud cases.

The standard alternative to accuracy is the precision-recall family of metrics, evaluated per class and then aggregated.

For a class $c$:

$$\text{Precision}_c = \frac{\text{TP}_c}{\text{TP}_c + \text{FP}_c}$$ $$\text{Recall}_c = \frac{\text{TP}_c}{\text{TP}_c + \text{FN}_c}$$ $$F_1^c = \frac{2 \cdot \text{Precision}_c \cdot \text{Recall}_c}{\text{Precision}_c + \text{Recall}_c}$$

where:

• $\text{TP}_c$ (True Positives): examples correctly predicted as class $c$
• $\text{FP}_c$ (False Positives): examples of other classes incorrectly predicted as class $c$
• $\text{FN}_c$ (False Negatives): examples of class $c$ incorrectly predicted as another class

Precision measures how often the model is right when it fires on class $c$. Recall measures what fraction of the true class $c$ examples the model retrieves. F1 is their harmonic mean, penalizing models that sacrifice one to maximize the other. Note that Precision and Recall trade off: raising the decision threshold increases Precision but decreases Recall, and vice versa.

The harmonic mean matters here. If a model achieves 100% precision by predicting the positive class on only one example and getting it right, its F1 will be near zero because recall is near zero. If it achieves 100% recall by predicting positive for every example, its F1 will be low because precision will be near the base rate. F1 rewards balanced performance, which is usually what you want.

The macro-average F1 averages the per-class F1 scores with equal weight:

$$\text{Macro-F1} = \frac{1}{C} \sum_{c=1}^{C} F_1^c$$

The weighted-average F1 weights each class by its support (number of examples):

$$\text{Weighted-F1} = \frac{\sum_{c=1}^{C} n_c \cdot F_1^c}{\sum_{c=1}^{C} n_c}$$

where $n_c$ is the number of examples in class $c$. Macro-F1 is more informative for imbalanced datasets because it gives equal importance to rare classes. Weighted-F1 reflects overall system performance weighted by class frequency.

The choice between macro and weighted F1 is a business decision disguised as a technical one. Weighted-F1 tells you how the system performs on average traffic. Macro-F1 tells you how the system performs on each category equally. If the rare categories (fraud, critical safety issues) are the ones that matter most, macro-F1 is the right headline metric, even though it is sensitive to performance on categories with very few examples.

The classification report reveals what overall accuracy hides: the fraud detection class (report_fraud) has significantly lower recall than the other classes because it has fewer training examples and higher confusion with other intents. The overall accuracy of approximately 90% looks reasonable, but the fraud recall of approximately 75% means one in four fraud reports is miscategorized. Whether that is acceptable depends on the application's risk tolerance, but reporting accuracy alone would have obscured this critical gap entirely.

When recall and precision are not equally important, you can generalize F1 to weight one more heavily than the other using the $F_\beta$ metric:

$$F_\beta = (1 + \beta^2) \cdot \frac{\text{Precision} \cdot \text{Recall}}{(\beta^2 \cdot \text{Precision}) + \text{Recall}}$$

where:

• $\beta > 1$: weights recall more heavily than precision (useful when missing positives is costly, as in fraud or medical screening)
• $\beta < 1$: weights precision more heavily than recall (useful when false positives are costly, as in content recommendation or legal document review)
• $\beta = 1$: the standard F1 metric, treating precision and recall equally

In fraud detection, you might use $F_2$ to ensure the metric penalizes misses more than false alarms. In spam filtering, where users strongly object to legitimate email being marked as spam, you might use $F_{0.5}$ to penalize false positives more. The $\beta$ parameter makes the business tradeoff explicit and encodes it directly into the metric.

The key modeling parameters for text classification systems are:

• model: The pre-trained backbone to fine-tune. distilbert-base-uncased-finetuned-sst-2-english is fast and suitable for binary sentiment. For zero-shot classification, cross-encoder/nli-deberta-v3-small is compact and efficient; facebook/bart-large-mnli is larger and often more accurate but requires a 1.6GB download. For fine-tuning, bert-base-uncased or roberta-base are standard starting points.
• device: The compute backend for inference. -1 forces CPU; use 0 for GPU when available. GPU inference is typically 10-30x faster for transformer models.
• OOS_THRESHOLD: The confidence floor for out-of-scope rejection in intent detection. Values between 0.4 and 0.6 are typical; tune on a held-out OOS development set.
• ngram_range: For TF-IDF classifiers, (1, 2) includes unigrams and bigrams, capturing local negation patterns. Extending to (1, 3) rarely helps and increases feature space substantially.
• C (logistic regression): The inverse regularization strength. Smaller values produce stronger regularization. C=1.0 is a reasonable default; reduce to 0.1 for small datasets to prevent overfitting.
• class_weight: Set to 'balanced' in LogisticRegression or LinearSVC to automatically upweight rare classes proportional to their inverse frequency.