Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

ELECTRA › Using Pre-trained ELECTRA

Michael BrenndoerferUpdated July 25, 20252 min read

Language AI HandbookAll chapters

Part 24 of 67BERT and Variants

Chapter 175ELECTRA › Using Pre-trained ELECTRA

Reading section 13 of 18 · Using Pre-trained ELECTRA · 2 minSection 13 of 18 · 2 min

Section 13 of 18 · 2 min read

ELECTRA › Using Pre-trained ELECTRAOne section · Sections in this chapter13 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

In practice, you'll load pre-trained ELECTRA models from Hugging Face rather than training from scratch:

In[33]:

Code

from transformers import ElectraModel, ElectraTokenizer

# Load pre-trained ELECTRA
tokenizer = ElectraTokenizer.from_pretrained(
    "google/electra-small-discriminator"
)
model = ElectraModel.from_pretrained("google/electra-small-discriminator")
model.eval()

# Prepare input
text = "ELECTRA learns efficiently by detecting replaced tokens."
inputs = tokenizer(text, return_tensors="pt")

Out[34]:

Console

Input text: ELECTRA learns efficiently by detecting replaced tokens.
Token IDs: [101, 11322, 2527, 10229, 18228, 2011, 25952, 2999, 19204, 2015, 1012, 102]
Tokens: ['[CLS]', 'elect', '##ra', 'learns', 'efficiently', 'by', 'detecting', 'replaced', 'token', '##s', '.', '[SEP]']

The tokenizer adds special tokens ([CLS] at the start, [SEP] at the end) and splits words using WordPiece. Notice that "efficiently" becomes "efficient" + "##ly". The ## prefix indicates a subword continuation.

In[35]:

Code

# Get hidden states
with torch.no_grad():
    outputs = model(**inputs)
    hidden_states = outputs.last_hidden_state

Out[36]:

Console

Hidden states shape: torch.Size([1, 12, 256])
  - Batch size: 1, Sequence length: 12, Hidden dimension: 256

CLS embedding (first 10 dims): [-0.1344696432352066, 0.18001538515090942, -0.5841341614723206, 0.3974262475967407, -0.09176667034626007, 0.02052440494298935, 0.706288754940033, 1.0155177116394043, 0.9312738180160522, -0.17593993246555328]

The model outputs a 256-dimensional vector for each token (ELECTRA-Small's hidden size). The [CLS] token's representation at position 0 aggregates information from the entire sequence and is typically used for classification tasks.

For classification tasks, use the task-specific model directly:

In[37]:

Code

from transformers import (
    ElectraForSequenceClassification as HFElectraForSequenceClassification,
)

# For sentiment classification
classifier = HFElectraForSequenceClassification.from_pretrained(
    "google/electra-small-discriminator",
    num_labels=2,  # Binary classification
)
classifier.eval()

with torch.no_grad():
    outputs = classifier(**inputs)
    logits = outputs.logits
    probs = F.softmax(logits, dim=-1)

Out[38]:

Console

Classification output:
  Logits: [0.0374, -0.0296]
  Probabilities: [0.5167, 0.4833]
  Predicted class: 0

The probabilities are close to 50/50, which is expected: the classification head was randomly initialized and hasn't been fine-tuned on any labeled data. After fine-tuning on a sentiment dataset (like SST-2), the model would produce meaningful, confident predictions.

Note that the untrained classification head produces random outputs. After fine-tuning on a sentiment dataset, the model would produce meaningful predictions.