Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Attention Intuition › Attention in Practice: Sentiment Analysis

Michael BrenndoerferMay 16, 20253 min read

Language AI HandbookAll chapters

Part 12 of 67Sequence-to-Sequence

Chapter 86Attention Intuition › Attention in Practice: Sentiment Analysis

Reading section 12 of 20 · Attention in Practice: Sentiment Analysis · 3 minSection 12 of 20 · 3 min

Section 12 of 20 · 3 min read

Attention Intuition › Attention in Practice: Sentiment AnalysisOne section · Sections in this chapter12 / 20 ↓

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To see attention in a more complete context, let's build a simple attention-based classifier for sentiment analysis. This shows how attention identifies which words drive the prediction.

In[11]:

Code

import torch
import torch.nn as nn
import torch.nn.functional as F


class AttentionClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(
            embed_dim, hidden_dim, batch_first=True, bidirectional=True
        )

        # Attention: single linear layer produces score for each position
        self.attention_score = nn.Linear(hidden_dim * 2, 1)

        # Classifier over context vector
        self.classifier = nn.Linear(hidden_dim * 2, num_classes)

    def forward(self, x):
        embedded = self.embedding(x)  # (batch, seq_len, embed_dim)
        lstm_out, _ = self.lstm(embedded)  # (batch, seq_len, hidden_dim * 2)

        # Compute attention scores and normalize
        scores = self.attention_score(lstm_out).squeeze(-1)  # (batch, seq_len)
        attention = F.softmax(scores, dim=1)  # (batch, seq_len)

        # Compute context vector: weighted sum of LSTM outputs
        context = torch.bmm(attention.unsqueeze(1), lstm_out).squeeze(1)

        logits = self.classifier(context)
        return logits, attention

This model uses a bidirectional LSTM to encode the input, then applies attention to create a single context vector for classification. The attention weights tell us which words the model considers most important. Notice the key architectural detail: the attention scoring function here is parameterized by a single linear layer (self.attention_score) that maps each hidden state to a scalar. This is called "content-only attention" because the score depends only on the hidden state itself, not on any external query. It is commonly used for classification tasks where there is no explicit query vector. Unlike seq2seq decoding, where the decoder state provides the query, classification simply needs to know which positions carry the most task-relevant information.

Let's create a small vocabulary and test the model:

In[12]:

Code

words = [
    "<pad>",
    "<unk>",
    "the",
    "movie",
    "was",
    "great",
    "terrible",
    "acting",
    "plot",
    "boring",
    "excellent",
    "not",
    "really",
    "loved",
    "hated",
]
word_to_idx = {w: i for i, w in enumerate(words)}


def tokenize(text):
    tokens = text.lower().split()
    return [word_to_idx.get(t, word_to_idx["<unk>"]) for t in tokens]


torch.manual_seed(42)
model = AttentionClassifier(
    vocab_size=len(words), embed_dim=32, hidden_dim=64, num_classes=2
)

In[13]:

Code

test_sentences = [
    "the movie was great",
    "the movie was terrible",
    "the acting was not great",
    "really loved the plot",
]

model.eval()
results = []

for sentence in test_sentences:
    tokens = tokenize(sentence)
    x = torch.tensor([tokens])

    with torch.no_grad():
        logits, attention = model(x)

    results.append(
        {
            "sentence": sentence,
            "tokens": sentence.lower().split(),
            "attention": attention[0].numpy(),
        }
    )

Out[14]:

Console

Attention weights for test sentences:

Sentence: "the movie was great"
Attention distribution:
  the       : 0.235 xxxxxxx
  movie     : 0.243 xxxxxxx
  was       : 0.241 xxxxxxx
  great     : 0.281 xxxxxxxx
  Highest attention: 'great' (0.281)

Sentence: "the movie was terrible"
Attention distribution:
  the       : 0.240 xxxxxxx
  movie     : 0.247 xxxxxxx
  was       : 0.244 xxxxxxx
  terrible  : 0.270 xxxxxxxx
  Highest attention: 'terrible' (0.270)

Sentence: "the acting was not great"
Attention distribution:
  the       : 0.186 xxxxx
  acting    : 0.190 xxxxx
  was       : 0.186 xxxxx
  not       : 0.218 xxxxxx
  great     : 0.220 xxxxxx
  Highest attention: 'great' (0.220)

Sentence: "really loved the plot"
Attention distribution:
  really    : 0.256 xxxxxxx
  loved     : 0.263 xxxxxxx
  the       : 0.236 xxxxxxx
  plot      : 0.244 xxxxxxx
  Highest attention: 'loved' (0.263)

Since this is an untrained model with randomly initialized weights, the attention distribution appears arbitrary. After training on labeled sentiment data, we would expect sentiment-bearing words like "great", "terrible", "loved", and "hated" to receive substantially higher attention weights (0.5 to 0.8), while function words like "the" and "was" would receive minimal attention (0.02 to 0.10).