Longformer › Worked Example: Tracing Attention Through a Document

To make the abstract mechanics concrete, let's trace what happens when Longformer processes a short legal document. Suppose the document has 20 tokens representing the sentence: "[CLS] The buyer shall pay the seller within 30 days of delivery as specified in clause 5 of this agreement [SEP]". We use a window size of 4 and designate position 0 ([CLS]) as the only global token.

The first thing to establish is which positions each token can attend to. Token 0 ([CLS]) is global, so it attends to all 20 positions. Every other token also attends back to position 0, even if it is outside their local window. Token 1 ("The") can attend to positions 0 through 3 (the local window plus the out-of-window global token at position 0 is already covered since the window extends to position 0 here). Token 5 ("pay") can attend to positions 3 through 7 under its local window, plus position 0 because it is global. Token 14 ("clause") can attend to positions 12 through 16 locally, plus position 0 globally.

Now consider a question: does the word "days" (token 10) have any connection to the phrase "clause 5" (tokens 14-15)? Under pure sliding window attention with window size 4, token 10 can only see positions 8 through 12, so it has no direct view of tokens 14-15. However, both token 10 and tokens 14-15 attend to the [CLS] token at position 0. In the first layer, the [CLS] token aggregates a weighted combination of all 20 tokens. In the second layer, when token 10 attends to the [CLS] token, it receives a representation that has already absorbed information from "clause 5". The long-range connection is established in two attention hops rather than requiring the sequence to be short enough for direct attention.

This two-hop communication is the mechanism behind Longformer's ability to model long-range dependencies despite using a local attention window for most tokens. Notice that the quality of this connection depends heavily on the [CLS] token learning to preserve the relevant information about "clause 5" in its aggregated representation. This is why training with global tokens on appropriate tasks matters: the model must learn to use global tokens as effective information relay stations, which happens through exposure to tasks that require cross-document reasoning.

In practice, real documents are far longer than 20 tokens, and the two-hop path becomes even more valuable. A 4,096-token legal contract might require reasoning between a definition in section 1 and its application in section 8. Under pure local attention with window size 512, reaching from position 100 to position 3,800 would require at least 7 layers of propagation. With global attention on section headers or the [CLS] token, that connection becomes a two-hop path available from the very first layer.