Bidirectional RNNs › Applications: Sequence Labeling Tasks

Bidirectional processing was designed for and excels at sequence labeling, where the model assigns a label to every token in the input.

In NER, the model reads a sentence and assigns each word a label such as B-PER (beginning of a person entity), I-PER (inside a person entity), B-ORG, I-ORG, O (outside any entity), and so on, following the BIO scheme covered in Part VI.

A BiLSTM is particularly well-suited for NER because entity boundaries often depend on both preceding and following context. The surname "Jordan" could be a person, a country, or a river. The context before it might disambiguate nothing (subject position is neutral), but the context after it, such as "scored 38 points" versus "borders Iraq", makes the entity type unambiguous.

A standard BiLSTM-CRF model for NER passes the BiLSTM output through a CRF layer that enforces globally consistent label sequences. The BiLSTM handles the contextual representation; the CRF handles the structured output constraint (you cannot have an I-PER tag immediately after an O tag, for example). This combination, introduced by Lample et al. in 2016, became the dominant NER architecture before BERT.

The division of labor here is clean and important. The BiLSTM produces, at each position, a vector of scores over all possible tags: how likely is each of B-PER, I-PER, B-ORG, I-ORG, O, and so on? These scores are locally informed but globally unconstrained. The CRF layer then finds the globally optimal tag sequence over all positions jointly, subject to learned transition costs between tags. It can encode knowledge like "I-LOC can only follow B-LOC or I-LOC, never O or B-PER." This global inference step catches inconsistencies that would slip through a per-position softmax. For instance, the per-position softmax might confidently predict I-ORG for position 3 even if position 2 was tagged O, because it only sees position 3 in isolation. The CRF prevents this by considering the whole sequence at once.

The BiLSTM-CRF paper reported state-of-the-art results on the CoNLL 2003 NER benchmark with a model that also incorporated character-level representations. Character-level BiLSTMs were used to encode each word's spelling. This provides morphological information (prefixes, suffixes, capitalization) that word-level embeddings miss. The spelling of "McCartney" provides strong evidence for a person entity even without context, but a word-level model would not know this for an out-of-vocabulary name it has never seen. The combination of word embeddings, character-level BiLSTMs, sentence-level BiLSTMs, and a CRF output layer formed a complete system that dominated NER research for several years.

POS tagging assigns grammatical categories (noun, verb, adjective, preposition, determiner, etc.) to each word. Like NER, it benefits from seeing both sides of a word. The sentence "They can fish in the lake" requires knowing that "can" is followed by "fish" (another potential noun or verb) and that "fish" is followed by "in the lake" (a prepositional phrase indicating location, not the action of swimming). The backward pass on a BiRNN provides exactly this right-side context before any forward-direction decisions are made.

POS tagging also illustrates an important property of bidirectionality: it improves the accuracy of ambiguous tokens and the confidence of unambiguous ones. Even when the forward context is sufficient to determine the correct tag, having the backward context as corroborating evidence produces a more confident and stable representation. This reduces variance across different sentences and leads to better generalization.

Noun phrase chunking, covered in Part VI, also benefits from bidirectionality. Determining where a chunk begins and ends requires both knowing what came before (whether we are inside a noun phrase) and what comes next (whether the next token opens a new phrase or continues the current one).

Chunking is essentially a boundary detection problem layered on top of sequence labeling. Each token receives a label indicating its role within a chunk: B (beginning of chunk), I (inside chunk), or O (outside all chunks). The precise definition of chunk boundaries often requires seeing several tokens in both directions simultaneously. For example, identifying the end of a noun phrase in "the very large blue car near the corner" is much easier with backward context: "near" signals the end of the first noun phrase and the beginning of a new prepositional phrase.

More broadly, any task where the correct output for a token depends on the surrounding sequence rather than just the prefix benefits from bidirectionality. This covers a wide swath of classic NLP pipelines: semantic role labeling, coreference mention detection, relation extraction, and even reading comprehension tasks where spans must be identified within a passage. The common thread is that the full context is available at inference time, so there is no reason to artificially restrict the model to left context only.

Semantic role labeling (SRL) assigns argument roles to spans in a sentence: who is the agent (the entity doing something), what is the patient (the entity being acted upon), what is the instrument, and so on. For the sentence "The chef in the kitchen prepared the meal with fresh ingredients," SRL must identify "The chef in the kitchen" as the agent of "prepared," "the meal" as the patient, and "fresh ingredients" as the instrument.

SRL is especially challenging for unidirectional models because the predicate (the main verb) often appears in the middle of the sentence, and arguments can appear both before and after it. A model that has only seen tokens up to the predicate has no information about what arguments follow it. A BiLSTM has full information about both preceding and following arguments when computing the representation of the predicate, which makes argument identification significantly more accurate.

Coreference resolution identifies when different expressions in a text refer to the same entity. Determining that "she" in the second sentence of a paragraph refers to "Dr. Nguyen" introduced in the first sentence requires reading forward through the text. But within a single sentence, the antecedent for a pronoun often appears later: "After she finished the experiment, Dr. Nguyen wrote up the results." Here "she" refers to "Dr. Nguyen," which appears after the pronoun. A unidirectional model reading left-to-right would have no information about "Dr. Nguyen" when it processes "she." A BiLSTM's backward pass has already processed "Dr. Nguyen" before encoding "she," making the coreference link much easier to identify.