Part-of-Speech Tagging › Tag Sets

A tag set defines the inventory of labels a tagger assigns. The choice of tag set involves a fundamental tradeoff: more tags allow finer-grained distinctions that benefit some downstream tasks, but require more annotated data to learn reliably and are harder to assign consistently.

The Penn Treebank (PTB) tag set, developed at the University of Pennsylvania as part of the Penn Treebank project in the early 1990s, became the standard for English NLP. It contains 36 POS tags plus 12 punctuation and symbol tags.

The fine-grained verb distinctions matter considerably in practice. Knowing that "barked" is VBD (past tense) versus "barking" is VBG (gerund/participle) tells a parser whether to attach it as the main verb or as a modifier. The six verb tags capture the full English verbal paradigm and allow downstream systems to reason about tense and aspect.

The Penn Treebank tag set was built around the Wall Street Journal corpus and reflects the characteristics of formal written English. This origin has consequences: the tag set handles financial text well but struggles with informal registers, social media text, and domain-specific terminology.

One design decision in the PTB tag set deserves particular attention: the treatment of "IN" as a single tag covering both prepositions and subordinating conjunctions. This conflation is linguistically controversial. "After dinner" uses "after" as a preposition governing a noun phrase, while "after she left" uses "after" as a subordinating conjunction introducing a clause. These are syntactically distinct, but they were collapsed into a single tag because reliable disambiguation requires the kind of deep syntactic analysis the tag set was meant to enable in the first place. This circular dependency is a recurring theme in annotation design: the tags you want to assign depend on information that the tags are supposed to provide.

The PTB tag set is designed for English, and extending it to other languages is problematic. German uses case inflections that English lacks. Mandarin Chinese has no morphological inflection at all. Arabic has a complex system of verb derivations. The Universal Dependencies (UD) project introduced a cross-linguistic tag set with 17 UPOS tags designed to apply consistently across languages.

The 17-tag set achieves cross-linguistic applicability by collapsing distinctions that are English-specific. All six English verb inflection tags (VB, VBD, VBG, VBN, VBP, VBZ) merge into VERB. The key insight is that morphological distinctions like tense can be captured through additional morphological annotation layers (called morphological features in UD) rather than proliferating the POS tag set itself.

The UD project has annotated over 200 treebanks across more than 100 languages using this framework, making it the largest cross-linguistic POS resource available.

The UD tag set also makes principled decisions about categories that vary typologically. The AUX (auxiliary) tag separates auxiliary verbs from main verbs, a distinction that matters in many languages. ADP covers both prepositions (as in English) and postpositions (as in Japanese, Turkish, and many other languages), recognizing that the functional role is the same regardless of word order. The PART tag handles particles that do not fit elsewhere, like the English infinitive marker "to" in its verbal use, or the negative particle in languages that express negation with a separate uninflected word.

The right tag set depends on your application. Here is how to think about it:

• Use Penn Treebank tags when working on English NLP with existing tools like NLTK or spaCy in English mode, or when downstream tasks (like parsing) benefit from fine-grained distinctions
• Use Universal Dependencies tags when building multilingual systems, working with non-English languages, or when compatibility across language resources matters more than granularity
• Use custom tags only when your domain has structures neither system handles well (e.g., specialized clinical or legal notation)

Most production systems today use one of these two standards. The Penn Treebank standard dominates English-only pipelines; Universal Dependencies dominates multilingual ones.

The question of granularity is worth thinking about carefully. Finer tag sets can express more information, but they are harder to learn reliably because each individual tag sees fewer training examples. A tag set with 100 labels means that each label appears, on average, in only 1% of tokens. Rare labels may have so few training examples that the model cannot learn their distributional patterns at all. The Penn Treebank's 36 tags strike a balance that proved empirically useful: fine enough to serve syntactic parsers, coarse enough to learn with moderate-sized corpora.