Global Tokens › Worked Example: Tracing Information Through a Document
Language AI HandbookAll chapters Global Tokens › Worked Example: Tracing Information Through a DocumentOne section · Sections in this chapter7 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To make the mechanics concrete, let us trace how a specific piece of information travels through a Longformer-style model processing a 20-token document. We will use a single global CLS token and a window size of 4.
Consider this document (indexed 0-19): "The cat [CLS] sat on the mat near the old red barn where the white horse lived yesterday."
Wait, that framing is slightly off: the CLS token is always at position 0 in Longformer, and the actual text tokens follow. Let us say the document is: [CLS] The cat sat on the mat near the old red barn where the white horse lived yesterday . (positions 0 through 17).
Suppose we want to understand how the final token yesterday at position 17 can be influenced by the word cat at position 2. With a window size of 4, yesterday (position 17) directly sees positions 15 through 17 and the CLS token at position 0. It cannot directly see position 2.
In layer 1, cat at position 2 attends to positions 0 through 4 (its local window) plus the CLS token. It contributes to the CLS token's representation by being in the CLS token's attention scope (since CLS attends to all positions). So after layer 1, the CLS token's representation has absorbed information from cat.
In layer 2, yesterday at position 17 attends to its local window (positions 15-17) and the CLS token. The CLS token now carries information derived from cat. So yesterday's representation after layer 2 is shaped by cat's information, mediated through the CLS token. The path is: cat (layer 1) to CLS, then CLS (layer 2) to yesterday. Two hops, two layers.
Let us verify this computationally:
# Worked example: trace information flow in a 18-token document
doc_len = 18
doc_window = 4
doc_global = [0] # CLS at position 0
cat_pos = 2
yesterday_pos = 17
# Layer 1 mask
layer1_mask = create_longformer_attention_mask(doc_len, doc_window, doc_global)
# Can 'cat' reach 'CLS' in layer 1?
cat_can_see_cls_l1 = layer1_mask[cat_pos, 0]
cls_can_see_cat_l1 = layer1_mask[0, cat_pos] # CLS has global attention
# Can 'yesterday' reach 'CLS' in layer 1?
yesterday_can_see_cls_l1 = layer1_mask[yesterday_pos, 0]
# After layer 2: yesterday has seen CLS, which has seen cat
# So information path: cat -> CLS (layer 1) -> yesterday (layer 2)
path_exists_in_2_layers = cls_can_see_cat_l1 and yesterday_can_see_cls_l1=== Worked Example: 18-token document, window=4, CLS at position 0 === Target path: 'cat' (pos 2) -> 'yesterday' (pos 17) Direct distance: 15 positions Layer 1 direct connections: cat (pos 2) can see CLS: True CLS can see cat (pos 2): True yesterday (pos 17) can see CLS: True After 2 layers, 'cat' can reach 'yesterday': True Positions visible to each key token in layer 1: 'cat' (pos 2) attends to: [0, 1, 2, 3, 4] 'yesterday' (pos 17) attends to: [0, 15, 16, 17] 'CLS' (pos 0) attends to: all 18 positions
The output confirms the 2-hop path. In layer 1, cat is in the CLS token's global attention scope. This contributes to CLS's updated representation. In layer 2, yesterday attends to CLS, which now encodes information from cat. The 15-position gap between cat and yesterday is bridged in exactly 2 layers.
The guarantee has a practical consequence: a Longformer model with only 2 transformer layers could, in principle, model dependencies between any two tokens regardless of distance, provided those dependencies route through the global token. In practice, models use many more layers to build rich, multi-hop reasoning chains through the global tokens.
