Classification Applications: Sentiment, Intent, and Scale

Text classification sits at the center of nearly every language AI deployment. When a customer support platform routes a ticket to the billing team instead of the technical team, that is classification. When a content moderation system flags a post for review, that is classification. When a voice assistant interprets "set a reminder for 7 AM" as a calendar intent rather than an alarm intent, that is classification. The operation is deceptively simple: assign a label from a predefined set to a piece of text. The challenge is doing it reliably, at speed, across the full messy diversity of real-world language.

Classification is one of the oldest tasks in NLP, and it remains one of the most economically important. Modern transformer-based models have changed the accuracy ceiling and the entire paradigm for how you approach classification. In the pre-BERT era, building a reliable classifier required careful feature engineering, curated lexicons, and thousands of labeled examples. Today, a large language model can classify text zero-shot with a natural language prompt, fine-tune to competitive performance with a few hundred examples, or be adapted through prompting alone when no training data exists. The toolkit has expanded dramatically, but so have the scale requirements and the subtlety of the problems being solved.

This chapter covers four pillars of applied text classification: sentiment analysis, topic classification, intent detection, and classification at scale. Each has its own design considerations, evaluation pitfalls, and architectural tradeoffs. We will work through the concepts, the math, and the code, starting from the intuitions that make these problems tractable and building to the engineering choices that make them production-ready.

Sentiment analysis is the task of identifying the emotional polarity of text: positive, negative, or neutral at minimum, with richer taxonomies extending to emotions, aspects, and intensity. It is arguably the most studied classification task in NLP, with roots in early computational linguistics and a direct line to modern transformer-based systems.

The commercial appetite for sentiment analysis is enormous. Financial firms analyze earnings call transcripts to predict stock movements. Consumer brands track product reviews to identify quality issues before they surface in returns. Political campaigns monitor social media to gauge voter sentiment in real time. The demand exists because language encodes opinion at a scale that human readers cannot process manually, and because the signal is valuable when extracted reliably.

Sentiment operates at multiple granularities, and the right level depends on the application.

Document-level sentiment assigns a single polarity to an entire document. A five-star review is positive. A scathing op-ed is negative. A neutral news report is, well, neutral. This level is conceptually straightforward but informationally lossy: a product review that praises battery life while criticizing the camera produces a mixed signal that document-level analysis flattens into a single score. Many products in the consumer electronics space receive an average three-star rating not because they are mediocre in all dimensions, but because they excel in some and disappoint in others. Document-level analysis treats both types of product identically, which limits its actionability.

Sentence-level sentiment decomposes a document into individual claims and assigns polarity to each. This is more granular but introduces its own challenges: negation, sarcasm, and discourse structure mean that sentences do not carry polarity in isolation. "I would never say this product is good" is negative despite containing the word "good." Understanding this requires the model to track the negation operator "never" and recognize that it inverts the sentiment of the clause that follows. Discourse markers further complicate things: "The camera is great, but the battery life is terrible" contains both positive and negative polarity in a single sentence connected by an adversative conjunction.

Aspect-based sentiment analysis (ABSA) is the most sophisticated level. It identifies specific aspects of an entity, such as product features or service dimensions, and assigns sentiment to each aspect independently. A review might be positive on "durability" and negative on "price/performance" simultaneously. ABSA is the industrially useful level for product analytics, customer satisfaction tracking, and competitive analysis. It requires more complex models and labeled data, but the output is actionable in ways that document-level scores are not: you can tell the product team that battery life is a pain point while build quality is praised, and that message lands differently than "your product has a 3.2/5 average rating."

Understanding what drives a sentiment classifier helps you reason about its failure modes. Early lexicon-based approaches worked from curated word lists: "excellent," "outstanding," and "terrific" are positive; "terrible," "awful," and "broken" are negative. The sentiment score was computed by aggregating polarity weights across the words in a document.

The problem with lexicon-based approaches is that sentiment is contextual. "This battery lasts an unpredictably short time" is negative, but "unpredictable" is not in most negative lexicons. "This laptop is sick" was negative in 1990 and positive in 2020. Irony and sarcasm invert the polarity of literally positive words: "Oh great, another bug" is clearly negative. No static word list captures these dynamics.

Statistical classifiers learned over n-gram features improved on lexicons by capturing local context: the bigram "not good" encodes negation, the trigram "could be better" encodes hedged dissatisfaction. But the feature space remained sparse and brittle for long-range dependencies. A negation phrase at the start of a long sentence ("I cannot say enough good things about...") can span twenty or thirty tokens before reaching the sentiment-bearing words, and bag-of-ngrams models have no mechanism to link them.

Pre-trained transformer models changed the calculation. A model like BERT trained on massive corpora has already seen billions of examples of humans expressing sentiment in context. Fine-tuning on a labeled sentiment dataset requires the model to adjust its already-rich representations rather than learn polarity from scratch. The result is a classifier that understands "sick" as a compliment in tech product reviews, that handles negation through attention over long spans, and that generalizes to phrasings it has never seen labeled.

The mathematical mechanism in a fine-tuned transformer is straightforward. Given input tokens $x_1, \ldots, x_n$, the transformer produces a sequence of contextualized embeddings $\mathbf{h}_1, \ldots, \mathbf{h}_n$. For classification, the representation at the special [CLS] token is extracted:

where the index $[0]$ selects the first position in the output sequence, corresponding to the [CLS] token prepended to the input. This representation is then projected to logits over the label set:

where:

• $\mathbf{W} \in \mathbb{R}^{C \times d}$: the classification head weight matrix, mapping from the hidden dimension $d$ to the number of classes $C$
• $\mathbf{b} \in \mathbb{R}^{C}$: the classification head bias
• $\hat{y} \in \mathbb{R}^{C}$: the predicted probability distribution over classes

The model is trained by minimizing cross-entropy loss between $\hat{y}$ and the one-hot encoded ground-truth label $y$:

This is the same cross-entropy loss we encountered in earlier chapters on language modeling, applied here to a fixed-size label set rather than the full vocabulary. The key insight is that fine-tuning modifies only the classification head and a small fraction of the transformer weights. Most of the work is already done: the pre-trained model's representations already encode the semantic and syntactic structure that distinguishes positive from negative language. The fine-tuning step teaches the model which aspects of those representations correspond to the sentiment categories in your specific label schema.

So far we have treated sentiment as a categorical variable, but sentiment also has intensity. "Acceptable" and "exceptional" are both positive, but they differ in degree. "Disappointed" and "infuriated" are both negative, but they differ substantially. For some applications, a continuous sentiment score is more useful than a discrete label.

The standard way to produce a continuous score from a classification model is to interpret the softmax probabilities as a weighted combination. For a three-class model with classes negative ($c=0$), neutral ($c=1$), and positive ($c=2$), you can compute a sentiment score in $[-1, 1]$ as:

where:

• $\hat{y}_2$: the predicted probability of positive sentiment
• $\hat{y}_0$: the predicted probability of negative sentiment

This score is $+1$ when the model is fully confident in positive, $-1$ when fully confident in negative, and $0$ when the model is neutral or when positive and negative probabilities cancel. A more refined variant weights by the full probability vector:

where $w_c$ is a numeric weight assigned to class $c$ ($-1$ for negative, $0$ for neutral, $+1$ for positive). This approach naturally extends to five-class or seven-class ordinal schemas by choosing weights to reflect the intensity ordering.

Alternatively, you can train a regression model directly on numeric ratings (such as 1-5 star reviews) rather than discretizing them into sentiment categories. The same transformer backbone applies, replacing the softmax classification head with a linear output that predicts a scalar value. Mean squared error or mean absolute error is the training objective. Regression-based models capture the full ordinal structure of the data and are especially appropriate when the downstream application needs graded scores rather than category labels.

Achieving reliable sentiment analysis in production requires confronting several recurring challenges.

Domain shift is the most common failure mode. A model trained on movie reviews does not transfer cleanly to customer support tickets. The vocabulary, the sentence structure, the distribution of sentiment expressions, and even the label definition can all differ. A "neutral" movie review might say "serviceable but not remarkable." A "neutral" support ticket might say "I need help with my account." Fine-tuning on domain-specific data is usually necessary, and the amount of labeled data required depends on how far the target domain sits from the model's pre-training distribution.

Implicit sentiment occurs when polarity is conveyed without explicit sentiment words. "I've returned the product twice" expresses negative sentiment without any emotional vocabulary. "The shipping took three weeks" implies disappointment without any evaluative language. Detecting implicit sentiment requires inference beyond surface patterns, which is where transformer representations excel relative to lexicon-based methods, but still present a real challenge for models fine-tuned primarily on explicit examples.

Negation and hedging continue to trip up classifiers. "I don't dislike it" is mildly positive despite containing "don't" and "dislike." "It might be good for some people" is non-committal despite "good." Handling these constructions correctly requires the model to integrate multiple spans of context, which attention mechanisms support in principle but do not guarantee in practice.

Label subjectivity is a measurement problem. Sentiment labels are assigned by humans, and humans disagree. Annotator agreement on three-class sentiment tasks (positive, negative, neutral) typically falls in the 0.6-0.8 kappa range, meaning 20-40% of examples are ambiguous. A model that learns from noisy labels will itself be noisy, and evaluation metrics that treat the test set as ground truth are optimistic about real-world performance.

A confusion matrix makes these failure modes concrete. The off-diagonal cells reveal systematic confusions: positive reviews mistaken for neutral and negative reviews mistaken for neutral are the most common errors, because the boundary between strong sentiment and hedged sentiment is uncertain.

The neutral class shows the lowest recall in this example, a consistent finding across sentiment datasets. Texts that express mild preference, hedged satisfaction, or mixed opinions sit in a ambiguous region where even human annotators disagree. The model's errors on neutral examples are not really failures; they reflect the fundamental fuzziness of the three-class schema.

This is a practical lesson for system design: if your application requires high accuracy on neutral examples, you may need to either refine the neutral definition with clearer annotation guidelines, collapse the schema to binary (positive vs. not-positive), or accept that neutral will always be the weak class and design your downstream logic accordingly.

Topic classification assigns a document to one or more thematic categories: politics, sports, technology, health. It is structurally similar to sentiment analysis (both are text classification tasks), but the nature of the signal is different. Sentiment is about emotional polarity; topic is about content domain. A document about climate change legislation is definitively in the "politics" category regardless of whether the author supports or opposes the policy.

The application space for topic classification is broad. News aggregators route articles to category pages. Email clients filter newsletters from support requests from promotional offers. Academic databases organize papers into subject areas. Document management systems tag contracts by type. In each case, the goal is to assign content to a category based on its thematic content rather than its surface form.

One of the most consequential decisions in topic classification is the label schema: what categories exist, how many there are, and how they are defined. This is usually treated as a given in benchmark evaluations but is a live engineering decision in production systems.

A flat label schema assigns each document to one of $K$ categories. For a news classification system, those categories might be: World, Sports, Business, Technology, Entertainment, Science, Health. The choice of $K$ involves a tradeoff between specificity (more categories capture finer distinctions) and coverage (more categories mean more training data per category required). Empirically, classifiers trained on more than 50-100 categories often see performance drops for rare categories that appear fewer than a few hundred times in the training set. Data volume alone does not explain this problem. When you have 100 categories, the decision boundaries are close together in representation space, and a model needs many examples per category to learn fine-grained distinctions between adjacent classes.

A hierarchical label schema recognizes that topics have natural structure. "Machine Learning" is a subcategory of "Technology," which is a subcategory of "Science and Technology." Hierarchical classification can exploit this structure in two ways: by training a cascaded classifier that first predicts the parent category and then the child, or by incorporating the taxonomy into the loss function to penalize near-misses less than complete misses. A cascaded system is simpler to implement and allows you to tune each level independently, but errors at the top level propagate to the bottom. A flat classifier trained with taxonomy-aware loss is end-to-end but requires careful loss design.

Multi-label classification handles documents that belong to multiple categories. An article about a political leader's health belongs to both Politics and Health. Multi-label classification relaxes the assumption of mutual exclusivity, treating each label as a binary decision rather than a one-from-$K$ selection. The architecture is the same transformer backbone, but the final layer uses per-class sigmoid activations instead of a single softmax, and the training loss is a sum of binary cross-entropies. We return to this in detail in the Classification at Scale section.

The key difference between topic and sentiment classifiers is the nature of the discriminative signal. Sentiment classifiers primarily rely on evaluative vocabulary and syntactic constructions such as negation, hedging, and intensifiers. Topic classifiers primarily rely on content vocabulary and entity references.

This has practical implications for data requirements. Topic classification benefits enormously from pre-trained representations because content vocabulary is broad but structured: the word "polymerase" reliably predicts the Biology topic, "quarterback" predicts Sports, "referendum" predicts Politics. A pre-trained language model has already learned rich representations of these domain-specific terms from its training corpus. Fine-tuning on a topic classification dataset lets the model organize these representations into a discriminative classifier with relatively few labeled examples.

Sentiment classification is somewhat more universal in its vocabulary (emotional words appear across domains), but aspect-level and domain-specific sentiment requires domain adaptation. The practical data requirements for topic classification are thus often lower than for domain-specific sentiment analysis, assuming the target categories are well-covered in the model's pre-training distribution.

There is a deeper reason why pre-training helps so much for topic classification. Topics correspond to coherent semantic fields: the Biology topic is characterized by the word "polymerase" and by a whole network of co-occurring terms ("gene," "protein," "sequence," "expression"), named entities (journals, institutions, researchers), and writing conventions. A pre-trained model encodes this entire field as a dense region of representation space. Telling it "Biology" during fine-tuning provides a label for a semantic region that already exists in the model's representations, rather than asking the model to construct that region from labeled examples alone. This is why topic classifiers often achieve strong performance with surprisingly little labeled data.

Modern large language models open a particularly useful capability for topic classification: zero-shot inference using natural language hypothesis testing. This approach, popularized by the Natural Language Inference (NLI) formulation of zero-shot classification, requires no labeled training data at all.

The insight is to repurpose an NLI model for classification. An NLI model takes a premise-hypothesis pair and predicts whether the hypothesis is entailed by, contradicted by, or neutral to the premise. To classify whether a document belongs to a topic, you construct a hypothesis: "This text is about sports." If the NLI model assigns high probability to entailment, the document is in the Sports category.

For a document $d$ and a set of candidate labels $\{l_1, \ldots, l_K\}$, zero-shot classification computes:

where:

• $P_{\text{NLI}}(\text{entailment} \mid \cdot, \cdot)$: the entailment probability from the NLI model
• $t(l_k)$: a natural language template instantiated with label $l_k$, such as "This text is about $l_k$" or "The topic of this text is $l_k$"

The assigned label is the one with the highest entailment probability:

Zero-shot classification is especially valuable when:

• The category set is new or changes frequently
• Labeled training data is scarce or expensive to collect
• The classification task is exploratory (you are still figuring out the right categories)

The accuracy is generally below a fine-tuned specialist model, but for many topic classification tasks the gap is smaller than intuition suggests. Zero-shot classifiers using large NLI models often achieve accuracy within 5-10 percentage points of fine-tuned models on standard benchmarks, without any task-specific labels.

The quality of the natural language template matters more than it might seem. A template like "This text is about $l_k$" works well for concrete nouns ("sports," "technology") but poorly for abstract or overlapping categories ("opinion," "analysis"). Descriptive templates often outperform terse ones: "The main topic of this article is $l_k$" and "The author is discussing $l_k$ in this passage" can produce different entailment scores for the same text. In practice, trying two or three template variants on a small validation set and selecting the best one is a cost-effective way to improve zero-shot accuracy without any labeled training data.

Another important design choice is the label name itself. "Tech" and "Technology" produce different entailment probabilities. More descriptive category names, such as "consumer technology products" or "scientific research," often outperform short abbreviations by giving the NLI model more linguistic signal about what the category contains. This is the zero-shot equivalent of writing a good label description: the more informative the label text, the better the model can reason about whether the document entails it.

Real-world topic classification systems often require going beyond the standard flat single-label setup. Understanding the two most common extensions helps you choose the right architecture.

Hierarchical classification is motivated by taxonomy structures that appear naturally in many domains. The ArXiv academic paper taxonomy has broad areas (Computer Science, Mathematics, Physics) subdivided into dozens of specializations. A medical record system might classify notes by organ system (Cardiology, Neurology, Oncology) and then by condition type within each system. Hierarchical approaches tend to perform better than flat approaches when there are many fine-grained categories, because the hierarchy provides structure that constrains where errors can occur. A document that is truly about Cardiology and is mistakenly classified as Pulmonology is a smaller error than one classified as Oncology, and hierarchy-aware loss functions can encode this intuition directly.

Multi-label classification is necessary when the assumption of mutual exclusivity fails. A news article covering a politician's medical treatment legitimately belongs to both Politics and Health. A research paper introducing a new algorithm might span Computer Science and Mathematics. When you enforce single-label classification on multi-label data, you force the model to pick one label and suppress its signal about the others. The multi-label architecture, which we cover in detail in the Classification at Scale section, handles this gracefully by treating each label as a separate binary prediction.

Intent detection is the problem of identifying what a user wants to accomplish from a natural language utterance. It is the classification task that powers conversational AI: when a user says "What's the weather like tomorrow?", intent detection identifies the WeatherQuery intent. When they say "Cancel my subscription," it identifies the CancelSubscription intent. The detected intent then drives downstream action: querying a weather API, opening an account management flow, triggering a confirmation dialog.

Intent detection is structurally identical to topic classification, but the characteristics of the task are quite different. Where topic classification deals with documents ranging from sentences to pages, intent detection operates on short utterances, typically under 20 words. Where topic categories are broad thematic domains, intents are narrow, action-oriented categories defined by the application's capability set. And where topic classifiers deal with a relatively stable label set, intent detection systems must handle a fundamental tension between the fixed set of supported intents and the unbounded diversity of user expression.

The intent schema is the enumeration of all actions the system can take. For a banking assistant, it might include: CheckBalance, TransferFunds, PayBill, ReportFraudulentCharge, CloseAccount, OpenAccount. The number of intents varies widely across applications: voice assistants may support thousands, domain-specific bots may support fewer than 20.

Designing the intent schema well requires thinking carefully about the granularity of actions. If you define intents too broadly, the downstream action logic has insufficient information to proceed. "AccountManagement" as a single intent gives no guidance about what the user wants to do with their account. If you define intents too narrowly, the number of categories explodes and each category has too few training examples. A useful rule of thumb is that each intent should correspond to a distinct action that the system takes, and that action should differ in its effect from the actions triggered by adjacent intents.

A critical consideration that separates intent detection from generic topic classification is out-of-scope (OOS) detection: identifying when a user's intent falls outside the supported capability set. A user asking "Can you recommend a good restaurant?" to a banking assistant is expressing a valid intent, but one the system cannot fulfill. A naive intent classifier will assign this to the closest in-domain intent, perhaps ReportFraudulentCharge if the model picks up some spurious pattern, producing a confusing response. OOS detection adds either a special OOS class or a threshold-based rejection mechanism.

The threshold approach defines a confidence floor: if $\max_k P(l_k \mid d) < \tau$ for some threshold $\tau$, the system treats the utterance as out-of-scope. Choosing $\tau$ involves a tradeoff between false positives (in-scope utterances incorrectly rejected) and false negatives (OOS utterances incorrectly assigned to an in-domain intent). In practice, calibrating $\tau$ on a held-out OOS development set is essential: the default softmax probabilities from a fine-tuned classifier are often overconfident, assigning high probability to the nearest in-domain intent even for utterances far from the training distribution.

The plot below illustrates how the false positive rate (legitimate queries incorrectly rejected) and false negative rate (OOS queries incorrectly handled as in-scope) vary as you sweep the threshold from 0 to 1.

The optimal threshold depends on the cost structure of your application. If failing to catch an OOS query costs more than occasionally rejecting a legitimate one, as in a safety-critical system, you push the threshold up, accepting higher false positives to reduce false negatives. If annoying users with unnecessary rejections is your primary concern, you push it down. Neither end of the spectrum is universally correct: the threshold is a business decision as much as a modeling one.

An alternative to threshold-based OOS detection is to train an explicit OOS class using examples of out-of-scope utterances collected from real traffic or synthesized by paraphrasing unrelated tasks. The explicit class approach tends to work better when you have enough OOS examples to characterize the distribution, because the model learns what OOS looks like rather than simply looking for low confidence. The threshold approach is simpler and does not require OOS labels, making it the practical default for new deployments where OOS data is not yet available.

Short utterances present specific challenges for classification models. A tweet, a search query, or a voice command might be four words long. The signal-to-noise ratio in short text is low: every word matters, there is little context to resolve ambiguity, and the feature space is sparse.

Transformers handle short text better than earlier sequential models because self-attention allows the model to consider all pairwise interactions between tokens regardless of their distance. But very short sequences do not give the model much to work with. "Call her" is ambiguous: it could mean PhoneContact, ReadMessage, or something else entirely depending on context the utterance does not provide. Contextual intent detection systems that maintain conversation history address this by feeding prior turns into the model, allowing "Call her" to be resolved against "I need to reach my sister."

Another strategy for short-text classification is data augmentation: generating paraphrases of short training examples to expose the model to more surface variation. Large language models are effective paraphrase generators. Given "What's my balance?", a language model can generate "How much money do I have?", "Show me my account balance", "Can you check my balance?", and dozens of variants. Training on these augmented examples improves performance across lexical variations without requiring human annotation of the additional examples.

Intent detection rarely operates in isolation in production systems. Real conversational AI systems perform joint intent detection and entity extraction: the model simultaneously identifies the intent (BookFlight) and the slot values that parameterize the action (departure city: "New York", destination city: "London", date: "next Friday").

This joint modeling problem can be formulated as a multi-task classification and sequence labeling task. The model produces:

1. A distribution over intents from the [CLS] representation (classification)
2. A label for each token indicating whether it is part of a slot value, and which slot (sequence labeling, typically using BIO tagging)

Joint modeling generally outperforms pipeline systems (intent detection followed by separate slot filling) because the two tasks share information. The word "fly" in "I want to fly to Paris" is evidence for both the BookFlight intent and the existence of a destination slot. A joint model can exploit this co-occurrence, while a pipeline model must perform each step in isolation.

As we discussed in the sequence labeling chapter, BIO tagging assigns each token a label indicating whether it is the Beginning, Inside, or Outside of a named span. In the slot filling context, the tags carry additional information: B-DESTINATION, I-DESTINATION, B-DATE, O, etc.

The shared representation between intent and slot prediction creates a beneficial interaction. Consider the utterance "Book a flight to Paris on Friday." The word "Paris" provides strong evidence for the BookFlight intent (since booking flights involves destinations) and simultaneously is the value of the DESTINATION slot. In a pipeline system, the slot filler would only see "Paris" after the intent has been decided, missing the opportunity to use the slot evidence when making the intent prediction. In a joint model, the transformer attends over the full utterance simultaneously, allowing each prediction to inform the other. This mutual reinforcement is why joint models outperform pipelines, particularly on ambiguous utterances where neither the intent nor the slot values are obvious from a partial view of the text.

Training a joint model requires labeled data in a specific format: each utterance must carry both an intent label and token-level slot annotations. Public datasets like ATIS (Airline Travel Information System) and SNIPS provide this format. When you lack labeled data for a new domain, zero-shot and few-shot approaches based on generative models can be used to bootstrap the joint model, though performance is lower than fully supervised training.

Let's work through practical implementations of each classification type. We will use the transformers library and scikit-learn to demonstrate both fine-tuned and zero-shot approaches.

We start by installing the required libraries and preparing sample datasets.

We have a balanced sentiment dataset with positive, negative, and neutral examples and an intent dataset covering four in-scope banking intents plus out-of-scope examples. The balance matters: imbalanced training data produces classifiers biased toward majority classes. Even in this toy dataset, the label distribution signals something important: in real deployments, out-of-scope traffic often represents 30-50% of all queries, which means your OOS detection system is handling a major portion of actual user requests, not a rare edge case.

For sentiment analysis, we will use a pre-trained sentiment model from Hugging Face rather than fine-tuning from scratch, since this illustrates how production deployments often work: use an existing model, test it on your domain, fine-tune only if needed.

The model handles direct positive and negative sentiment well. The negation and sarcasm examples are harder: "I would never say this product is good" expresses neutrality or mild negativity, and "Oh great, another software update..." is sarcastic. These are exactly the cases where lexicon-based approaches fail catastrophically, and even neural models require substantial examples of such constructions to learn them reliably. The SST-2 training data consists of movie review snippets, which contain fewer sarcastic constructions than, say, Twitter data. This illustrates domain shift in practice: the model's training domain shapes which failure modes it exhibits on out-of-domain text.

To understand what the transformer model is learning, it helps to compare it against a simple TF-IDF baseline. The baseline makes its decision criteria explicit: you can inspect which words contribute most to each class.

The TF-IDF model reveals the lexical patterns that drive each class: "brilliant," "outstanding," and "masterpiece" for positive; "terrible," "waste," and "cringeworthy" for negative. These are real signals, but they are surface-level. The model has no mechanism to understand that "not good" is negative or that "sick" can be a compliment. You can use this feature inspection as a diagnostic tool: if the top features for a class contain surprising words, that reveals annotation artifacts or domain-specific conventions in your training data.

For topic classification, we demonstrate the zero-shot NLI approach using cross-encoder/nli-deberta-v3-small, a compact DeBERTa-based model designed for natural language inference. This requires no labeled training data and works by framing classification as textual entailment.

Zero-shot classification assigns the correct category to each article without any task-specific training. The confidence scores reflect the model's uncertainty: clear-cut cases like the political article or the sports article receive high confidence scores, while ambiguous cases that span multiple categories receive more distributed scores. Notice that the neural network architecture article might receive partial scores for both "technology" and "science," since the boundary between these categories is blurry from a linguistic perspective.

This is informative in its own right. When zero-shot scores are distributed across multiple labels, that is often a sign that the document is legitimately ambiguous or multi-topic, not a failure of the model. You can use the distribution of scores as a soft multi-label assignment: if both "technology" and "science" receive scores above 0.3, the document may warrant both labels.

Intent detection requires special handling of out-of-scope utterances. We implement a threshold-based rejection mechanism.

The threshold mechanism correctly identifies out-of-scope queries by their lower confidence scores. In-scope utterances receive high confidence scores because the model's NLI training allows it to recognize that "What's my account balance?" entails the "check_balance" intent description. The nearest intent output for OOS cases is diagnostically useful during development: it tells you which in-scope intent the model found most relevant, which can reveal confusions in your intent schema or gaps in your OOS coverage.

The preceding examples work well as prototypes. Taking classification systems to production at scale introduces a set of engineering and modeling challenges that do not appear in tutorials.

A large language model producing a single classification requires the same forward pass as generating text: all attention heads, all feed-forward layers, full context window. For document classification in an offline batch pipeline, this is acceptable. For real-time intent detection in a voice assistant that must respond within 200 milliseconds, it is not.

The standard architecture for high-throughput classification uses knowledge distillation to compress a large teacher model into a smaller student model that preserves most of the accuracy at a fraction of the compute cost. DistilBERT, for example, has 40% fewer parameters than BERT-base while retaining 97% of its performance on GLUE benchmarks. For a classification task, you can push further: a 3-layer DistilBERT or a task-specific student model can often achieve acceptable accuracy at 10x lower latency than the full BERT model.

For very high throughput requirements, the classification head can be replaced entirely with a lightweight model trained on the frozen embeddings from a large encoder. The large encoder runs once per document to produce a fixed-size embedding; the lightweight model (logistic regression, a shallow neural network) then classifies from that embedding at negligible cost. This is efficient when the same document appears multiple times (you can cache its embedding) or when the classification needs to be run multiple times with different label sets.

A third optimization that production teams use is quantization: converting the model's floating-point weights from 32-bit to 8-bit integers (INT8) or even 4-bit representations. INT8 quantization reduces memory footprint by 4x and often improves inference speed by 1.5-2x on modern hardware, with accuracy losses typically under 1% for classification tasks. Quantization is especially attractive because it can be applied post-training without any additional labeled data, making it accessible even when retraining is not an option.

The choice between these approaches depends on your latency budget and accuracy requirements. For a voice assistant with 200ms total response time, you might need all three: a distilled 3-layer student model, INT8 quantized, with frozen embeddings cached for frequent queries. For a batch content moderation pipeline processing documents overnight, the full BERT model may be perfectly acceptable and the distillation investment not worth making. Map your latency requirements to your deployment context before optimizing prematurely.

Most real-world classification datasets are imbalanced. In a customer support classification system, the vast majority of tickets might be account inquiries, with fraud reports making up 0.1% of the volume. A naive classifier that predicts "account inquiry" for everything achieves 99.9% accuracy but is completely useless for the task that matters.

The standard toolkit for class imbalance includes:

• Class weights: modify the cross-entropy loss to penalize errors on rare classes more heavily. The weight for class $c$ is typically set inversely proportional to its frequency: $w_c = N / (C \cdot N_c)$, where $N$ is the total number of examples, $C$ is the number of classes, and $N_c$ is the count of class $c$.
• Oversampling: replicate examples from rare classes in the training batch. SMOTE (Synthetic Minority Oversampling Technique) generates synthetic examples by interpolating between existing rare-class examples in feature space.
• Threshold adjustment: instead of using 0.5 as the decision threshold for binary classification, set class-specific thresholds based on the desired precision-recall tradeoff for each class.
• Evaluation on the right metrics: accuracy is misleading for imbalanced data. Report per-class precision, recall, F1, and the macro-average F1 (unweighted average across classes). A system that handles the rare critical class poorly will show this failure clearly in per-class metrics even if overall accuracy is high.

The class weight approach is the simplest and most widely applicable. It requires no changes to the data or training loop beyond passing a class_weight argument to your optimizer or loss function, and it directly addresses the objective: telling the model that errors on rare classes cost more. It works well when the imbalance is moderate (10:1 or less). When the imbalance is extreme (100:1 or worse), oversampling rare classes is often necessary in addition to weighting, because a very rare class may appear so infrequently in each epoch that the model's gradient signal from that class is negligible even with high weights.

Many real-world classification problems are multi-label: a news article can be simultaneously about Sports and Health (an athlete's injury), or about Technology and Business (a startup acquisition). Multi-label classification requires treating each label as an independent binary decision.

The standard approach replaces the final softmax with per-class sigmoid activations:

where:

• $\hat{y}_c \in [0, 1]$: the predicted probability that the document belongs to class $c$
• $\mathbf{w}_c$: the weight vector for class $c$ in the classification head
• $\sigma(\cdot)$: the sigmoid activation function, which maps each logit to an independent probability

The training loss becomes a sum of binary cross-entropies across all classes:

where $y_c \in \{0, 1\}$ is the ground-truth binary indicator for class $c$. Unlike the single-label case, the ground-truth vector can have multiple ones.

At inference time, a threshold $\tau$ (often 0.5) determines which classes are assigned:

where:

• $\hat{l}_c \in \{0, 1\}$: the predicted binary label for class $c$
• $\hat{y}_c$: the sigmoid probability output for class $c$
• $\tau$: the decision threshold, typically 0.5 but tunable per class
• $\mathbb{1}[\cdot]$: the indicator function, returning 1 when the condition is satisfied and 0 otherwise

Per-class threshold tuning is especially important in multi-label settings: a class that rarely appears alongside others should have a lower threshold than one that co-occurs frequently, since the prior probability of each class assignment is independent. A practical procedure is to sweep thresholds per class on a validation set and select the one that maximizes the per-class F1. This requires a validation set with multi-label ground truth, but the tuning itself is inexpensive.

One subtlety in multi-label classification is that labels are not truly independent. "Sports" and "Health" co-occur (athlete injuries, sports medicine), while "Sports" and "Politics" co-occur less often. The per-class sigmoid approach ignores these co-occurrence patterns: each label is predicted independently from the same shared representation.

For applications where label co-occurrence patterns are strong and predictable, there are architectures that explicitly model label dependencies. Classifier chains predict one label at a time and condition each prediction on all previously predicted labels. Graph neural network approaches represent labels as nodes in a label correlation graph and propagate information between related labels before making predictions. These approaches can improve performance when label correlations are informative, but they add architectural complexity and are harder to debug. For most applications, the per-class sigmoid approach is the pragmatic starting point.

Expanding a classification system to new intents or topics typically requires new labeled data. Active learning reduces the labeling burden by selecting the most informative examples for human annotation.

The most widely used selection criterion is uncertainty sampling: the model labels a pool of unlabeled examples and selects for human review those where it is least confident. For classification, uncertainty can be measured by the Shannon entropy of the predicted distribution. A model that assigns equal probability to all $C$ classes (completely uncertain) has maximum entropy $\log C$. A model that assigns probability 1 to one class (completely certain) has entropy 0.

The entropy of a predicted probability distribution $\hat{y}$ over $C$ classes is:

where:

• $H(\hat{y}) \in [0, \log C]$: the entropy score, with 0 indicating certainty and $\log C$ indicating maximum uncertainty
• $\hat{y}_c$: the predicted probability for class $c$, with $\sum_c \hat{y}_c = 1$
• The negative sign ensures the result is non-negative, since $\log \hat{y}_c \leq 0$ for probabilities in $[0, 1]$

Higher entropy indicates greater uncertainty. Examples with high entropy are the ones where the model's predictions are most spread across classes, meaning they carry the most information for improving the decision boundaries.

In practice, active learning cycles proceed as follows: train an initial model on a seed set of labeled examples, run it on a large pool of unlabeled examples, select the $k$ examples with highest entropy, send those to human annotators, add the annotations to the training set, retrain, and repeat. Active learning consistently outperforms random sampling for the same annotation budget, often achieving the same accuracy with 30-50% fewer labeled examples.

Beyond uncertainty sampling, a complementary strategy is diversity sampling: selecting examples that are different from each other and from the existing training set. A classifier that is uncertain about 1,000 nearly identical examples does not benefit much from labeling all 1,000. Clustering the uncertain examples and selecting one representative per cluster gives you diverse coverage of the uncertainty region. In practice, combining uncertainty and diversity sampling (sometimes called core-set or BADGE approaches) yields the best results across a range of tasks.

Let's look at how classification confidence scores relate to accuracy, a concept called calibration. A well-calibrated classifier that assigns 70% confidence to predictions is correct about 70% of the time. A poorly calibrated classifier might assign 90% confidence to predictions it gets right only 60% of the time.

The reliability diagram shows the key difference between a well-calibrated and an overconfident model: the overconfident model assigns high confidence scores to predictions it gets wrong at a much higher rate. This matters operationally because many applications use confidence scores to trigger human review: a model that is always 90% confident will route few cases to human review even when it is wrong, while a well-calibrated model will express appropriate uncertainty on ambiguous cases.

The precision-recall curves show a characteristic pattern in imbalanced classification: the common class achieves a high area under the curve even with a simple model, while rare classes see significant precision-recall tradeoffs. For the rare class, high recall requires accepting lower precision (many false positives), and high precision requires accepting lower recall (many false negatives). The operating point on this curve is a product of the application's cost structure: a fraud detection system prefers high recall (catch most fraud even with some false alarms) while a clinical alert system might prefer high precision (only flag cases where the model is highly confident).

Let's visualize how different classification approaches distribute their confidence scores, an important diagnostic when choosing between methods.

The confidence distribution plot reveals a common pattern in practice: fine-tuned transformers are often overconfident, concentrating predictions near 1.0 even for correct predictions that the model should be uncertain about. Zero-shot NLI models show more spread because they rely on the NLI model's calibrated entailment probabilities. TF-IDF classifiers with logistic regression often sit in the middle, with sigmoid probabilities that reflect the model's distance from the decision boundary. When building systems that route low-confidence predictions to human review, verify that the confidence scores are informative by plotting reliability diagrams on a held-out evaluation set.

Classification evaluation requires care about which metrics you report and what they mean.

Accuracy, the fraction of correct predictions, is the most natural metric but fails in two important ways.

First, it treats all errors equally. Misclassifying a fraud report as an account inquiry has far higher cost than misclassifying an account inquiry as a fraud report. Applications with asymmetric error costs need weighted or asymmetric metrics.

Second, accuracy is misleading for imbalanced datasets. A classifier that always predicts the majority class in a 95/5 imbalance achieves 95% accuracy. That number is meaningless as an evaluation of the classifier's utility.

The combination of these two failures makes accuracy a particularly poor guide for systems where the economically important class is the rare one. In fraud detection, spam filtering, medical diagnosis, and content moderation, the class that matters is almost always the minority class. Reporting accuracy in these settings creates a false sense of security: a model with 98% accuracy can still be missing 40% of fraud cases.

The standard alternative to accuracy is the precision-recall family of metrics, evaluated per class and then aggregated.

For a class $c$:

where:

• $\text{TP}_c$ (True Positives): examples correctly predicted as class $c$
• $\text{FP}_c$ (False Positives): examples of other classes incorrectly predicted as class $c$
• $\text{FN}_c$ (False Negatives): examples of class $c$ incorrectly predicted as another class

Precision measures how often the model is right when it fires on class $c$. Recall measures what fraction of the true class $c$ examples the model retrieves. F1 is their harmonic mean, penalizing models that sacrifice one to maximize the other. Note that Precision and Recall trade off: raising the decision threshold increases Precision but decreases Recall, and vice versa.

The harmonic mean matters here. If a model achieves 100% precision by predicting the positive class on only one example and getting it right, its F1 will be near zero because recall is near zero. If it achieves 100% recall by predicting positive for every example, its F1 will be low because precision will be near the base rate. F1 rewards balanced performance, which is usually what you want.

The macro-average F1 averages the per-class F1 scores with equal weight:

The weighted-average F1 weights each class by its support (number of examples):

where $n_c$ is the number of examples in class $c$. Macro-F1 is more informative for imbalanced datasets because it gives equal importance to rare classes. Weighted-F1 reflects overall system performance weighted by class frequency.

The choice between macro and weighted F1 is a business decision disguised as a technical one. Weighted-F1 tells you how the system performs on average traffic. Macro-F1 tells you how the system performs on each category equally. If the rare categories (fraud, critical safety issues) are the ones that matter most, macro-F1 is the right headline metric, even though it is sensitive to performance on categories with very few examples.

The classification report reveals what overall accuracy hides: the fraud detection class (report_fraud) has significantly lower recall than the other classes because it has fewer training examples and higher confusion with other intents. The overall accuracy of approximately 90% looks reasonable, but the fraud recall of approximately 75% means one in four fraud reports is miscategorized. Whether that is acceptable depends on the application's risk tolerance, but reporting accuracy alone would have obscured this critical gap entirely.

When recall and precision are not equally important, you can generalize F1 to weight one more heavily than the other using the $F_\beta$ metric:

where:

• $\beta > 1$: weights recall more heavily than precision (useful when missing positives is costly, as in fraud or medical screening)
• $\beta < 1$: weights precision more heavily than recall (useful when false positives are costly, as in content recommendation or legal document review)
• $\beta = 1$: the standard F1 metric, treating precision and recall equally

In fraud detection, you might use $F_2$ to ensure the metric penalizes misses more than false alarms. In spam filtering, where users strongly object to legitimate email being marked as spam, you might use $F_{0.5}$ to penalize false positives more. The $\beta$ parameter makes the business tradeoff explicit and encodes it directly into the metric.

The key modeling parameters for text classification systems are:

• model: The pre-trained backbone to fine-tune. distilbert-base-uncased-finetuned-sst-2-english is fast and suitable for binary sentiment. For zero-shot classification, cross-encoder/nli-deberta-v3-small is compact and efficient; facebook/bart-large-mnli is larger and often more accurate but requires a 1.6GB download. For fine-tuning, bert-base-uncased or roberta-base are standard starting points.
• device: The compute backend for inference. -1 forces CPU; use 0 for GPU when available. GPU inference is typically 10-30x faster for transformer models.
• OOS_THRESHOLD: The confidence floor for out-of-scope rejection in intent detection. Values between 0.4 and 0.6 are typical; tune on a held-out OOS development set.
• ngram_range: For TF-IDF classifiers, (1, 2) includes unigrams and bigrams, capturing local negation patterns. Extending to (1, 3) rarely helps and increases feature space substantially.
• C (logistic regression): The inverse regularization strength. Smaller values produce stronger regularization. C=1.0 is a reasonable default; reduce to 0.1 for small datasets to prevent overfitting.
• class_weight: Set to 'balanced' in LogisticRegression or LinearSVC to automatically upweight rare classes proportional to their inverse frequency.

Text classification systems, despite their maturity as a technology, carry persistent limitations that shape their real-world reliability.

Distribution shift between training and deployment is the central challenge. A sentiment classifier trained on Amazon product reviews degrades when deployed to Twitter, where sentence fragments, slang, and sarcasm are far more common. An intent classifier trained on synthetic data generated by developers degrades when deployed to real users who phrase requests in ways the training set never anticipated. No amount of in-distribution performance optimization prepares a model for out-of-distribution queries. The only reliable mitigation is to collect real deployment data, monitor for performance drops, and retrain or adapt regularly.

The monitoring infrastructure for distribution shift is often underinvested. Many teams deploy a model, measure its performance on a test set at launch, and then consider the evaluation done. In practice, the query distribution shifts continuously as user behavior evolves, new topics emerge, and the product changes. Setting up a pipeline that samples a fraction of production traffic for human labeling, then running weekly or monthly model evaluations against those labeled samples, is the operational practice that turns a point-in-time evaluation into a continuous quality signal.

Label quality limits ceiling performance. The theoretical upper bound for a classifier is the agreement rate among human annotators on the same task. For fine-grained categories, that rate can be surprisingly low: inter-annotator agreement on fine-grained emotion classification often falls below 0.5 kappa. A model trained on such data cannot exceed human agreement in any meaningful sense. Investing in annotation quality, including clear label definitions, annotator training, and adjudication processes for disagreements, often produces larger gains than switching to a larger model.

Clear label definitions are particularly important and often neglected. "What distinguishes a Neutral tweet from a Positive one?" is a question that annotators will answer differently without explicit guidance. A two-page annotation guide with ten examples per category boundary, explaining common edge cases and how to handle them, can increase inter-annotator kappa from 0.55 to 0.75 for the same annotators and task. That improvement in label quality translates directly into a better model.

Zero-shot capabilities do not eliminate the need for evaluation. It is tempting to deploy a zero-shot classifier without evaluation because it requires no labeled data. But zero-shot performance varies dramatically across tasks, label sets, and domains. A zero-shot classifier that works well on news topics may fail badly on specialized medical categories. Before any deployment, you should construct at least a small evaluation set (100-200 examples covering all categories, including edge cases and OOS examples) and measure actual performance rather than relying on benchmark reports from different task configurations.

Multilingual classification presents distinct challenges. Most high-performing classifiers are trained primarily on English text. Cross-lingual transfer via multilingual models (mBERT, XLM-RoBERTa) has improved significantly, but performance for low-resource languages remains substantially below English performance, and the gap widens for fine-grained categories not well-represented in the pre-training corpus. Applications serving multilingual audiences should explicitly evaluate across all supported languages and plan for language-specific fine-tuning where accuracy gaps are unacceptable.

Temporal drift deserves particular attention in applications like social media monitoring and news classification. Topics evolve: "AI" in 2015 referred primarily to academic research; by 2023 it was a central business and political topic. Slang evolves: terms that express strong positive sentiment in one year can become ironic or neutral in the next. A classifier deployed without retraining will gradually drift away from the current distribution of language use, and the drift may be slow enough that no single evaluation catches the degradation. Periodic re-evaluation on fresh data, even a small amount, is the primary defense against temporal drift.

As we will see in the upcoming chapter on Conversational AI, intent detection is just one component of a conversational system. Handling multi-turn context, managing dialogue state, and generating appropriate responses introduce additional layers of complexity beyond the classification task itself.

Text classification is the workhorse of applied language AI. The four major application types, sentiment analysis, topic classification, intent detection, and large-scale classification pipelines, share a common transformer-based architecture but differ significantly in their label structures, failure modes, and operational requirements.

Key takeaways from this chapter:

• Sentiment analysis operates at multiple granularities (document, sentence, aspect) and requires domain-specific adaptation to handle implicit sentiment, sarcasm, and negation reliably. The [CLS] representation from a fine-tuned transformer encodes the contextual signal that lexicon-based methods miss.
• Topic classification benefits from zero-shot NLI-based approaches when training data is unavailable or when the label set is fluid, trading some accuracy for complete elimination of the annotation requirement. Label name quality and template design affect zero-shot accuracy.
• Intent detection requires explicit out-of-scope handling. A classifier without OOS rejection will always return an in-domain label regardless of how inappropriate the utterance is. Joint intent and entity modeling outperforms pipeline approaches because the two tasks share evidence.
• Classification at scale demands attention to throughput (knowledge distillation for latency), class imbalance (class weights, threshold adjustment, macro-F1 for evaluation), and distribution shift (continuous monitoring and retraining).
• Calibration deserves attention alongside accuracy. A classifier whose confidence scores reflect true accuracy enables better downstream decisions (human review routing, multi-stage pipelines) than one that is overconfident.
• Label quality is often the binding constraint on accuracy, not model size. Before scaling up to a larger model, ensure that annotation quality is high and that inter-annotator agreement is measured.
• Evaluation must go beyond overall accuracy to per-class precision, recall, and macro-F1. For applications with asymmetric error costs, $F_\beta$ with an appropriate $\beta$ encodes the business tradeoff directly into the metric.

The next chapter on Conversational AI will build on intent detection and extend the framework to multi-turn dialogue, slot filling, and dialogue management, showing how classification feeds into the broader architecture of interactive language systems.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about classification applications in language AI.