Classification Applications › Sentiment Analysis

Sentiment analysis is the task of identifying the emotional polarity of text: positive, negative, or neutral at minimum, with richer taxonomies extending to emotions, aspects, and intensity. It is arguably the most studied classification task in NLP, with roots in early computational linguistics and a direct line to modern transformer-based systems.

The commercial appetite for sentiment analysis is enormous. Financial firms analyze earnings call transcripts to predict stock movements. Consumer brands track product reviews to identify quality issues before they surface in returns. Political campaigns monitor social media to gauge voter sentiment in real time. The demand exists because language encodes opinion at a scale that human readers cannot process manually, and because the signal is valuable when extracted reliably.

Sentiment operates at multiple granularities, and the right level depends on the application.

Document-level sentiment assigns a single polarity to an entire document. A five-star review is positive. A scathing op-ed is negative. A neutral news report is, well, neutral. This level is conceptually straightforward but informationally lossy: a product review that praises battery life while criticizing the camera produces a mixed signal that document-level analysis flattens into a single score. Many products in the consumer electronics space receive an average three-star rating not because they are mediocre in all dimensions, but because they excel in some and disappoint in others. Document-level analysis treats both types of product identically, which limits its actionability.

Sentence-level sentiment decomposes a document into individual claims and assigns polarity to each. This is more granular but introduces its own challenges: negation, sarcasm, and discourse structure mean that sentences do not carry polarity in isolation. "I would never say this product is good" is negative despite containing the word "good." Understanding this requires the model to track the negation operator "never" and recognize that it inverts the sentiment of the clause that follows. Discourse markers further complicate things: "The camera is great, but the battery life is terrible" contains both positive and negative polarity in a single sentence connected by an adversative conjunction.

Aspect-based sentiment analysis (ABSA) is the most sophisticated level. It identifies specific aspects of an entity, such as product features or service dimensions, and assigns sentiment to each aspect independently. A review might be positive on "durability" and negative on "price/performance" simultaneously. ABSA is the industrially useful level for product analytics, customer satisfaction tracking, and competitive analysis. It requires more complex models and labeled data, but the output is actionable in ways that document-level scores are not: you can tell the product team that battery life is a pain point while build quality is praised, and that message lands differently than "your product has a 3.2/5 average rating."

Understanding what drives a sentiment classifier helps you reason about its failure modes. Early lexicon-based approaches worked from curated word lists: "excellent," "outstanding," and "terrific" are positive; "terrible," "awful," and "broken" are negative. The sentiment score was computed by aggregating polarity weights across the words in a document.

The problem with lexicon-based approaches is that sentiment is contextual. "This battery lasts an unpredictably short time" is negative, but "unpredictable" is not in most negative lexicons. "This laptop is sick" was negative in 1990 and positive in 2020. Irony and sarcasm invert the polarity of literally positive words: "Oh great, another bug" is clearly negative. No static word list captures these dynamics.

Statistical classifiers learned over n-gram features improved on lexicons by capturing local context: the bigram "not good" encodes negation, the trigram "could be better" encodes hedged dissatisfaction. But the feature space remained sparse and brittle for long-range dependencies. A negation phrase at the start of a long sentence ("I cannot say enough good things about...") can span twenty or thirty tokens before reaching the sentiment-bearing words, and bag-of-ngrams models have no mechanism to link them.

Pre-trained transformer models changed the calculation. A model like BERT trained on massive corpora has already seen billions of examples of humans expressing sentiment in context. Fine-tuning on a labeled sentiment dataset requires the model to adjust its already-rich representations rather than learn polarity from scratch. The result is a classifier that understands "sick" as a compliment in tech product reviews, that handles negation through attention over long spans, and that generalizes to phrasings it has never seen labeled.

The mathematical mechanism in a fine-tuned transformer is straightforward. Given input tokens $x_1, \ldots, x_n$, the transformer produces a sequence of contextualized embeddings $\mathbf{h}_1, \ldots, \mathbf{h}_n$. For classification, the representation at the special [CLS] token is extracted:

$$\mathbf{h}_{\text{cls}} = \text{Transformer}(x_1, \ldots, x_n)[0]$$

where the index $[0]$ selects the first position in the output sequence, corresponding to the [CLS] token prepended to the input. This representation is then projected to logits over the label set:

$$\mathbf{z} = \mathbf{W} \mathbf{h}_{\text{cls}} + \mathbf{b}$$ $$\hat{y} = \text{softmax}(\mathbf{z})$$

where:

• $\mathbf{W} \in \mathbb{R}^{C \times d}$: the classification head weight matrix, mapping from the hidden dimension $d$ to the number of classes $C$
• $\mathbf{b} \in \mathbb{R}^{C}$: the classification head bias
• $\hat{y} \in \mathbb{R}^{C}$: the predicted probability distribution over classes

The model is trained by minimizing cross-entropy loss between $\hat{y}$ and the one-hot encoded ground-truth label $y$:

$$\mathcal{L} = -\sum_{c=1}^{C} y_c \log \hat{y}_c$$

This is the same cross-entropy loss we encountered in earlier chapters on language modeling, applied here to a fixed-size label set rather than the full vocabulary. The key insight is that fine-tuning modifies only the classification head and a small fraction of the transformer weights. Most of the work is already done: the pre-trained model's representations already encode the semantic and syntactic structure that distinguishes positive from negative language. The fine-tuning step teaches the model which aspects of those representations correspond to the sentiment categories in your specific label schema.

So far we have treated sentiment as a categorical variable, but sentiment also has intensity. "Acceptable" and "exceptional" are both positive, but they differ in degree. "Disappointed" and "infuriated" are both negative, but they differ substantially. For some applications, a continuous sentiment score is more useful than a discrete label.

The standard way to produce a continuous score from a classification model is to interpret the softmax probabilities as a weighted combination. For a three-class model with classes negative ($c=0$), neutral ($c=1$), and positive ($c=2$), you can compute a sentiment score in $[-1, 1]$ as:

$$s = \hat{y}_2 - \hat{y}_0$$

where:

• $\hat{y}_2$: the predicted probability of positive sentiment
• $\hat{y}_0$: the predicted probability of negative sentiment

This score is $+1$ when the model is fully confident in positive, $-1$ when fully confident in negative, and $0$ when the model is neutral or when positive and negative probabilities cancel. A more refined variant weights by the full probability vector:

$$s = \sum_{c} w_c \cdot \hat{y}_c$$

where $w_c$ is a numeric weight assigned to class $c$ ($-1$ for negative, $0$ for neutral, $+1$ for positive). This approach naturally extends to five-class or seven-class ordinal schemas by choosing weights to reflect the intensity ordering.

Alternatively, you can train a regression model directly on numeric ratings (such as 1-5 star reviews) rather than discretizing them into sentiment categories. The same transformer backbone applies, replacing the softmax classification head with a linear output that predicts a scalar value. Mean squared error or mean absolute error is the training objective. Regression-based models capture the full ordinal structure of the data and are especially appropriate when the downstream application needs graded scores rather than category labels.

Achieving reliable sentiment analysis in production requires confronting several recurring challenges.

Domain shift is the most common failure mode. A model trained on movie reviews does not transfer cleanly to customer support tickets. The vocabulary, the sentence structure, the distribution of sentiment expressions, and even the label definition can all differ. A "neutral" movie review might say "serviceable but not remarkable." A "neutral" support ticket might say "I need help with my account." Fine-tuning on domain-specific data is usually necessary, and the amount of labeled data required depends on how far the target domain sits from the model's pre-training distribution.

Implicit sentiment occurs when polarity is conveyed without explicit sentiment words. "I've returned the product twice" expresses negative sentiment without any emotional vocabulary. "The shipping took three weeks" implies disappointment without any evaluative language. Detecting implicit sentiment requires inference beyond surface patterns, which is where transformer representations excel relative to lexicon-based methods, but still present a real challenge for models fine-tuned primarily on explicit examples.

Negation and hedging continue to trip up classifiers. "I don't dislike it" is mildly positive despite containing "don't" and "dislike." "It might be good for some people" is non-committal despite "good." Handling these constructions correctly requires the model to integrate multiple spans of context, which attention mechanisms support in principle but do not guarantee in practice.

Label subjectivity is a measurement problem. Sentiment labels are assigned by humans, and humans disagree. Annotator agreement on three-class sentiment tasks (positive, negative, neutral) typically falls in the 0.6-0.8 kappa range, meaning 20-40% of examples are ambiguous. A model that learns from noisy labels will itself be noisy, and evaluation metrics that treat the test set as ground truth are optimistic about real-world performance.

A confusion matrix makes these failure modes concrete. The off-diagonal cells reveal systematic confusions: positive reviews mistaken for neutral and negative reviews mistaken for neutral are the most common errors, because the boundary between strong sentiment and hedged sentiment is uncertain.