Part-of-Speech Tagging › Neural Approaches to POS Tagging
Language AI HandbookAll chapters Part-of-Speech Tagging › Neural Approaches to POS TaggingOne section · Sections in this chapter11 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The transition from statistical to neural approaches for POS tagging followed the broader trajectory of NLP. BiLSTM-based models appeared around 2015-2016 and substantially improved accuracy, particularly on unknown words. Transformer-based models (starting with BERT in 2018) pushed accuracy even higher by using large-scale pre-training on unlabeled text.
BiLSTM Taggers
A bidirectional LSTM (BiLSTM) tagger reads the input sentence in both forward and backward directions, producing a contextual representation for each token that depends on the entire sequence, rather than only a local window. The forward LSTM reads left-to-right, building up a hidden state that summarizes everything seen so far. The backward LSTM reads right-to-left, building a hidden state that summarizes everything from the right. Concatenating these two states produces a representation that is sensitive to context from both directions.
This bidirectionality is particularly valuable for POS tagging because POS ambiguity is often resolvable only with both left and right context. "I saw him" and "I saw a" both start with "I saw," but "him" (a pronoun) is a strong signal that "saw" is a past-tense verb, while "a" (a determiner introducing a noun phrase) is also consistent with that reading. However, "She picked up the saw" requires right context to confirm that "saw" is a noun: the next word "She" suggests a new clause was just completed, and picking something up implies a physical object.
The BiLSTM approach to POS tagging operates as follows. Each input word is first converted to a dense vector embedding (either a lookup embedding or a character-level encoding). The BiLSTM reads the sequence of embeddings and produces a contextual representation for each position. This representation is then passed through a linear layer followed by a softmax to produce a probability distribution over tags. The model is trained end-to-end with cross-entropy loss to predict the correct tag at each position.
Character-level encoding is important for handling unknown words. Rather than looking up a word in an embedding table (which fails for out-of-vocabulary words), a character-level CNN or LSTM reads the character sequence of the word and produces a representation that captures morphological patterns. A BiLSTM tagger that combines word embeddings with character-level encodings achieves much better unknown-word accuracy than one that uses only word embeddings.
BERT-Based Taggers
BERT (Bidirectional Encoder Representations from Transformers) changed POS tagging in the same way it changed every other NLP task: by providing contextual word representations of unprecedented quality, pre-trained on billions of words of text. Fine-tuning BERT on a POS-annotated dataset requires only a simple linear classification head on top of BERT's output representations.
The key advantage of BERT over BiLSTMs for POS tagging is the quality of its contextual representations. BERT was pre-trained to predict masked words in context, which forced it to learn deep syntactic and semantic patterns. Its representations encode the word's distributional properties and the way its meaning and syntactic role shift across contexts. When you ask a BERT-based tagger to tag "run" in "She goes for a morning run," the BERT representation of "run" already reflects the fact that it follows a determiner-adjective-noun sequence, making it much easier for the classification head to assign NOUN.
The practical workflow for BERT-based POS tagging uses the Hugging Face transformers library. The input sentence is tokenized using BERT's subword tokenizer, the subword tokens are passed through BERT to get contextual embeddings, and then a linear layer maps the embedding at each word position to a distribution over POS tags. One complication is that BERT's subword tokenization splits words into pieces (e.g., "running" might become "run" + "##ning"), so the tag assignment needs to be mapped back from subword positions to whole-word positions.
Here is a simplified illustration of what that workflow looks like:
# Illustrate the BERT subword alignment problem
sample_words = ["The", "researchers", "published", "groundbreaking", "findings"]
sample_tags = ["DET", "NOUN", "VERB", "ADJ", "NOUN"]
# Simulate what BERT's tokenizer does to these words
# (showing subword splits for multi-morpheme words)
subword_splits = {
"researchers": ["re", "##search", "##ers"],
"published": ["published"],
"groundbreaking": ["ground", "##breaking"],
"findings": ["find", "##ings"],
}
print("BERT Subword Alignment Challenge")
print("=" * 50)
print(f"\n{'Word':<20} {'Subwords':<35} {'Tag'}")
print("-" * 60)
for word, tag in zip(sample_words, sample_tags):
subwords = subword_splits.get(word, [word])
print(f"{word:<20} {str(subwords):<35} {tag}")
print("\nFor prediction: assign each word's tag to the FIRST subword token.")
print("The remaining subword tokens are masked out during loss computation.")BERT POS Tagging Alignment Strategy ================================================== Input word sequence: The researchers published groundbreaking findings Input tag sequence: DET NOUN VERB ADJ NOUN After BERT tokenization: [CLS] The re ##search ##ers published ground ##breaking find ##ings [SEP] Alignment: The --> token 1 --> predict DET researchers --> tokens 2-4 --> predict from token 2 (re-), ignore ##search, ##ers published --> token 5 --> predict VERB groundbreaking --> tokens 6-7 --> predict from token 6 (ground), ignore ##breaking findings --> tokens 8-9 --> predict from token 8 (find), ignore ##ings
The alignment strategy is to predict the tag from the first subword token of each word and ignore subsequent subword pieces during loss computation. This works well in practice because the contextual representation of the first subword piece already incorporates information from the entire word (through BERT's self-attention mechanism).