BERT Representations › The CLS Token: A Sentence in a Vector
Language AI HandbookAll chapters BERT Representations › The CLS Token: A Sentence in a VectorOne section · Sections in this chapter2 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BERT prepends a special [CLS] token to every input sequence. After passing through all transformer layers, this token's final hidden state is intended to represent the entire sequence. But why does shoving extra information into position zero work at all?
The key is self-attention. Every layer allows [CLS] to attend to all other tokens. Over 12 layers, information from the entire sequence flows into this position. The model learns during pre-training, through the Next Sentence Prediction task, to aggregate sentence-level meaning at [CLS].
Think of [CLS] as a reporter who attends every conversation. The reporter can listen to all participants simultaneously, ask follow-up questions across the whole room, and gradually builds a mental summary that captures the overall theme of the discussion. By the time the conversation ends, the reporter's notes reflect the gist of everything said. That summary is the [CLS] vector. Of course, a reporter's summary emphasizes some things and ignores others, and a summary is always a lossy compression of the original. The same is true for [CLS].
Notice that the [CLS] token starts with no information about the sentence. At the embedding layer, it holds only a learned position-zero embedding that BERT associated with sentence classification during pre-training. Its final representation comes entirely from what it learned to absorb through attention over all 12 layers. This is a learned compression process: BERT's pre-training rewarded the [CLS] vector whenever it could correctly predict whether two sentences were consecutive. That training signal shaped the [CLS] representation to encode sentence-level coherence and topic.
The [CLS] token design traces back to GPT's classification approach, where a special end-of-sequence delimiter's final hidden state was used for classification. The BERT authors placed this token at the beginning of the sequence rather than the end. Because BERT uses bidirectional attention, the position doesn't change what the token can attend to: a [CLS] token at position zero can still see every other token in the same forward pass. The NSP training objective gave this token a concrete reason to encode whole-sentence information, which made it a natural choice for classification heads in downstream fine-tuning.
The hidden state of the [CLS] token after the final transformer layer. During BERT pre-training, this representation is used for the Next Sentence Prediction objective, which encourages it to capture sentence-level semantics useful for binary classification.
Let's extract the [CLS] representation from a pre-trained BERT model.
from transformers import BertModel, BertTokenizer
# Load pre-trained BERT
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")
model.eval()
# Tokenize a sample sentence
text = "The quick brown fox jumps over the lazy dog."
inputs = tokenizer(text, return_tensors="pt")
# Get all hidden states
with torch.no_grad():
outputs = model(**inputs, output_hidden_states=True)
# The final layer's [CLS] representation
cls_embedding = outputs.last_hidden_state[0, 0, :] # [768]CLS Token Representation ---------------------------------------- Shape: torch.Size([768]) Mean: -0.0099 Std: 0.5359 Min: -6.9109 Max: 3.8243
The [CLS] vector is 768-dimensional for BERT-base. Most values sit near zero, with a small number of much larger positive and negative components. This single vector supposedly captures everything about our sentence.
The histogram reveals that most embedding dimensions have values between -0.5 and 0.5, with a roughly symmetric distribution. This structure emerges from BERT's layer normalization, which ensures stable training by keeping activations in a controlled range.
When CLS Works
The [CLS] representation works well for tasks that mirror BERT's pre-training objectives. Classification tasks, especially binary ones, align naturally with how [CLS] was trained through NSP.
In practice, the [CLS] token is the default choice for text classification pipelines built on BERT. When you fine-tune BERT for sentiment analysis, topic classification, or natural language inference, you typically attach a linear layer to the [CLS] output and train the whole system end-to-end. The combination works because gradient descent can reshape the [CLS] representation to separate classes, building on the general aggregation capacity the model developed during pre-training. The pre-training doesn't hand you a perfect sentence embedding; it gives you a vector that is already structured to receive a classification signal, which makes fine-tuning efficient.
def get_cls_representations(texts, model, tokenizer):
"""Extract [CLS] representations for a batch of texts."""
embeddings = []
for text in texts:
inputs = tokenizer(
text, return_tensors="pt", truncation=True, max_length=512
)
with torch.no_grad():
outputs = model(**inputs)
cls_emb = outputs.last_hidden_state[0, 0, :]
embeddings.append(cls_emb.numpy())
return np.array(embeddings)
# Example: sentiment analysis setup
positive_texts = [
"I loved this movie, it was fantastic!",
"Best experience of my life, highly recommend.",
"Absolutely wonderful, exceeded all expectations.",
]
negative_texts = [
"Terrible waste of time, completely boring.",
"I hated every minute of this disaster.",
"Worst purchase I ever made, avoid at all costs.",
]
pos_embeddings = get_cls_representations(positive_texts, model, tokenizer)
neg_embeddings = get_cls_representations(negative_texts, model, tokenizer)CLS Representation Clustering ---------------------------------------- Positive texts avg similarity: 0.9231 Negative texts avg similarity: 0.9382 Cross-class similarity: 0.8905 Within-class vs between-class gap: 0.0402
Even without fine-tuning, the [CLS] representations show some clustering by sentiment. Positive texts are more similar to each other than to negative texts. This separation, while modest, demonstrates that [CLS] captures semantic information relevant to classification.
When CLS Fails
The [CLS] token has a critical limitation: it was trained for NSP, not for general semantic similarity. For tasks like semantic search or sentence similarity, raw [CLS] embeddings perform surprisingly poorly.
The NSP objective is a weak training signal. It asks whether sentence B could plausibly follow sentence A in a real document. This is a binary prediction task requiring only that the model notice topical coherence, not that it represents the full meaning of each sentence in a geometrically useful way. A [CLS] vector trained on NSP learns to encode "this pair is coherent" or "this pair is random," but it doesn't learn to place semantically similar sentences near each other in vector space. When you compute cosine similarity between two [CLS] vectors, you're measuring something that correlates weakly at best with human judgments of sentence similarity.
Notice that this failure isn't a bug in BERT; it's a consequence of using a representation for a purpose it wasn't trained for. The model did exactly what it was asked to do during pre-training. The problem is that NSP's training signal didn't require the embedding space to have the geometric properties that similarity tasks need. This is why evaluating your representation strategy on a small held-out set before committing to a production pipeline is so important.
# Semantic similarity examples
similarity_pairs = [
("A man is playing guitar.", "Someone plays a musical instrument."),
("A man is playing guitar.", "A woman is cooking dinner."),
("The cat sleeps on the couch.", "A feline rests on the sofa."),
("The cat sleeps on the couch.", "The stock market crashed today."),
]
pair_embeddings = []
for s1, s2 in similarity_pairs:
emb1 = get_cls_representations([s1], model, tokenizer)[0]
emb2 = get_cls_representations([s2], model, tokenizer)[0]
sim = cosine_similarity([emb1], [emb2])[0, 0]
pair_embeddings.append((s1[:30], s2[:30], sim))CLS Cosine Similarities (Raw BERT) ------------------------------------------------------------ 'A man is playing guitar....' vs 'Someone plays a musical instru...' Similarity: 0.8996 'A man is playing guitar....' vs 'A woman is cooking dinner....' Similarity: 0.9625 'The cat sleeps on the couch....' vs 'A feline rests on the sofa....' Similarity: 0.9083 'The cat sleeps on the couch....' vs 'The stock market crashed today...' Similarity: 0.7857
The similarities are surprisingly high across the board, even for semantically unrelated sentences. This phenomenon, sometimes called the "anisotropy problem," occurs because BERT's [CLS] representations cluster in a narrow cone of the embedding space. All sentences end up relatively similar to each other, making it hard to distinguish truly related content from unrelated content.
The PCA projection makes the practical limitation visible: topic labels intermix instead of forming compact, separated clusters. The first two principal components summarize only part of the 768-dimensional variation, so this view is not a complete map of the embedding space. What it does show is that these raw [CLS] vectors do not separate the example topics cleanly.
This is why models like Sentence-BERT (SBERT) exist. They fine-tune BERT specifically for sentence similarity using contrastive or siamese objectives, producing representations where cosine similarity correlates with semantic relatedness. We'll examine SBERT in more depth in the chapter on dense retrieval and semantic search. For now, the takeaway is clear: if your task involves comparing sentences by meaning rather than classifying them, don't reach for raw [CLS] embeddings.