BERT Representations › BERT as a Feature Extractor
Language AI HandbookAll chapters BERT Representations › BERT as a Feature ExtractorOne section · Sections in this chapter5 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Using BERT as a feature extractor means taking its representations as fixed inputs to a downstream model. This contrasts with fine-tuning, where BERT's weights are updated during training.
Think of the feature extraction approach as hiring a skilled translator who has spent years studying a language. Instead of teaching them about your specific business domain, you ask them to convert every document into a standardized format, and then you build your analysis on top of their translations. The translator's skills are fixed; what you're customizing is the analysis layer, not the translation.
The historical motivation for feature extraction comes from the era before BERT, when contextual embeddings like ELMo were computationally expensive to run on large datasets. ELMo's original paper described using it primarily as a feature extractor: compute representations once, cache them, and feed them to task-specific models. BERT changed the economics of fine-tuning by making the full model accessible to practitioners with moderate GPU resources, but feature extraction remains valuable when inference latency and compute budget are constraints.
In practice, feature extraction is the right default when you're exploring a new task quickly. You can compare logistic regression, SVM, gradient boosting, and neural classifiers on the same BERT features in minutes, cycling through hypotheses without waiting for full model fine-tuning runs. Once you've identified a promising direction, you can decide whether fine-tuning on top of that approach is worth the additional investment.
The Feature Extraction Pipeline
Feature extraction treats BERT as a frozen encoder. You pass text through BERT once, save the embeddings, and train a separate classifier on those embeddings.
class BertFeatureExtractor:
"""
Extract features from BERT without fine-tuning.
"""
def __init__(self, model_name="bert-base-uncased", layer=-1, pooling="cls"):
self.tokenizer = BertTokenizer.from_pretrained(model_name)
self.model = BertModel.from_pretrained(model_name)
self.model.eval()
self.layer = layer # Which layer to use (-1 = last)
self.pooling = pooling
# Freeze all parameters
for param in self.model.parameters():
param.requires_grad = False
def extract(self, texts, batch_size=32):
"""Extract features for a list of texts."""
features = []
for i in range(0, len(texts), batch_size):
batch_texts = texts[i : i + batch_size]
inputs = self.tokenizer(
batch_texts,
return_tensors="pt",
padding=True,
truncation=True,
max_length=512,
)
with torch.no_grad():
outputs = self.model(**inputs, output_hidden_states=True)
# Get specified layer
hidden = outputs.hidden_states[self.layer]
mask = inputs["attention_mask"]
# Apply pooling
if self.pooling == "cls":
batch_features = hidden[:, 0, :]
elif self.pooling == "mean":
batch_features = mean_pooling(hidden, mask)
else:
raise ValueError(f"Unknown pooling: {self.pooling}")
features.append(batch_features.numpy())
return np.concatenate(features, axis=0)
# Example usage
extractor = BertFeatureExtractor(layer=-1, pooling="mean")
sample_texts = [
"This product is amazing!",
"Terrible experience, would not recommend.",
"It's okay, nothing special.",
]
features = extractor.extract(sample_texts)Feature Extraction Results ---------------------------------------- Number of texts: 3 Feature shape: (3, 768) Features per text: 768
These 768-dimensional features can now feed into any classifier: logistic regression, SVM, random forest, or a simple neural network. The most important property to note is that the features are precomputed. Once you've run BERT over your corpus and saved the output embeddings to disk, subsequent training runs don't require a GPU at all. You're just fitting a classifier to a matrix of numbers, which even a laptop CPU handles easily. This separation of the expensive inference step from the cheap classification step is the core practical advantage of the feature extraction pipeline.
Training a Classifier on Frozen Features
Let's train a simple classifier using extracted BERT features.
from sklearn.linear_model import LogisticRegression
# Create a small synthetic dataset for demonstration
# In practice, you'd have real labeled data
train_texts = [
"I love this product!",
"Best purchase ever",
"Highly recommended",
"Amazing quality",
"Fantastic experience",
"Terrible waste of money",
"Worst thing I bought",
"Complete disappointment",
"Avoid at all costs",
"Do not buy this",
"It's okay I guess",
"Nothing special about it",
]
train_labels = [1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0] # 1=positive, 0=negative
# Extract features
train_features = extractor.extract(train_texts)
# Train a simple classifier
clf = LogisticRegression(max_iter=1000, random_state=42)
clf.fit(train_features, train_labels)Simple Classifier on BERT Features ---------------------------------------- Training samples: 12 Feature dimensions: 768 Training accuracy: 100.00% Cross-validation accuracy: 83.33% (+/- 23.57%)
Even with a tiny dataset, the classifier achieves reasonable accuracy because BERT's pre-trained features already encode rich semantic information. The BERT encoder has seen hundreds of millions of sentences and learned that "I loved this" and "Absolutely wonderful" point toward similar regions of embedding space. The linear classifier's job is much easier than learning from scratch: it just needs to find the hyperplane that separates these pre-organized clusters.
Advantages of Feature Extraction
Using BERT as a frozen feature extractor offers several practical benefits:
- Speed: Extract features once, then train classifiers instantly. You don't need GPU access for every experiment
- Simplicity: Standard ML pipelines work directly. No need for gradient-based optimization of large models
- Low resource: Train on CPU with minimal memory. Fine-tuning BERT requires significant GPU memory
- Interpretability: Downstream models can be simpler and more interpretable (e.g., logistic regression with feature importance)
import time
def benchmark_approaches(texts, labels, n_iterations=5):
"""Compare feature extraction vs fine-tuning (simulated)."""
# Feature extraction approach
extractor = BertFeatureExtractor(pooling="mean")
start = time.time()
for _ in range(n_iterations):
features = extractor.extract(texts)
extraction_time = (time.time() - start) / n_iterations
# Training a classifier on frozen features is fast
start = time.time()
for _ in range(n_iterations):
clf = LogisticRegression(max_iter=100, random_state=42)
clf.fit(features, labels)
training_time = (time.time() - start) / n_iterations
return extraction_time, training_time
ext_time, train_time = benchmark_approaches(train_texts[:8], train_labels[:8])Timing Comparison (Feature Extraction) ---------------------------------------- Feature extraction: 0.042s per batch Classifier training: 0.0021s Total (extraction + training): 0.044s Note: Fine-tuning BERT would require: - Multiple epochs over the data - GPU acceleration - Much longer training time