LSTM Architecture › LSTM for Sequence Classification
Language AI HandbookAll chapters LSTM Architecture › LSTM for Sequence ClassificationOne section · Sections in this chapter8 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
One of the most common LSTM applications is classifying entire sequences: deciding the sentiment of a review, the intent of a query, or the topic of a document. The approach is to run the LSTM over the full sequence and use the final hidden state as a fixed-size summary vector for a linear classifier. The LSTM does the hard work of compressing a variable-length sequence into a fixed-size representation; the classifier on top is just a linear layer applied to that representation.
Building the Classifier
class LSTMClassifier(nn.Module):
"""LSTM sequence classifier using the final hidden state."""
def __init__(
self, input_size, hidden_size, num_classes, num_layers=1, dropout=0.0
):
super().__init__()
self.lstm = nn.LSTM(
input_size=input_size,
hidden_size=hidden_size,
num_layers=num_layers,
batch_first=True,
dropout=dropout if num_layers > 1 else 0.0,
)
self.classifier = nn.Linear(hidden_size, num_classes)
def forward(self, x):
# x: (batch, seq_len, input_size)
_, (hidden, _) = self.lstm(x)
# hidden[-1]: final layer's hidden state (batch, hidden_size)
return self.classifier(hidden[-1])The simplicity of this architecture is one of LSTMs' strengths for classification. You stack an LSTM on top of an embedding layer, apply the LSTM to get a sequence of hidden states, take the last hidden state, and pass it through a linear layer. The LSTM does all the work of summarizing variable-length sequences into fixed-size vectors.
A variation that often works better in practice uses the mean of all hidden states rather than just the last one. The final hidden state represents the network's processing state after the last token, which gives disproportionate weight to the end of the sequence. Averaging over all positions gives equal weight to information throughout the sequence. Max pooling across the time dimension is another common alternative: take the element-wise maximum across time, which captures the strongest activation for each hidden state dimension regardless of when it occurred. In practice, mean pooling and max pooling both tend to outperform using only the final hidden state on classification tasks, especially for long sequences.
Training and Evaluating
import torch.optim as optim
torch.manual_seed(42)
model_cls = LSTMClassifier(input_size=8, hidden_size=32, num_classes=2)
optimizer_cls = optim.Adam(model_cls.parameters(), lr=0.01)
criterion_cls = nn.CrossEntropyLoss()
def make_cls_batch(batch_size=32, seq_len=20, input_size=8):
"""Synthetic binary classification: class determined by sequence mean."""
x = torch.randn(batch_size, seq_len, input_size)
labels = (x.mean(dim=(1, 2)) + 0.3 * torch.randn(batch_size) > 0).long()
return x, labels
cls_losses = []
for step in range(100):
model_cls.train()
xb, yb = make_cls_batch()
optimizer_cls.zero_grad()
logits = model_cls(xb)
loss = criterion_cls(logits, yb)
loss.backward()
optimizer_cls.step()
cls_losses.append(loss.item())
model_cls.eval()
with torch.no_grad():
x_test, y_test = make_cls_batch(batch_size=200)
preds = model_cls(x_test).argmax(dim=1)
test_acc = (preds == y_test).float().mean().item()Initial loss: 0.7037 Final loss: 0.6739 Test accuracy: 58.5%
