Attention Visualization › Worked Example: Tracing "Bank" Disambiguation
Language AI HandbookAll chapters Attention Visualization › Worked Example: Tracing "Bank" DisambiguationOne section · Sections in this chapter8 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before scaling to systematic analysis, it helps to walk through one concrete example that illustrates both what attention visualization can reveal and its limits. The ambiguous word "bank" is a classic test case for disambiguation because the same surface form maps to two completely different meanings depending on context.
Consider processing "The bank can guarantee deposits will eventually cover future tuition costs" through BERT. You would like to know how the model resolves the meaning of "bank." The hypothesis: if BERT correctly understands that this is a financial bank, some attention head in a later layer should show "bank" attending to contextually disambiguating tokens like "deposits," "guarantee," or "tuition."
You can test this by extracting attention weights and checking whether there is a head in layers 5-12 where the token "bank" places substantial weight on "deposits." If you find such a head and confirm the pattern holds for other financial sentences but not for river-related sentences (e.g., "She walked along the bank of the river"), you have preliminary evidence of a disambiguation-related head.
But here is the problem: you can almost always find such a head if you look through all 144 attention matrices. With 144 matrices of different patterns for every sentence, some head will happen to show the pattern you are looking for. This is the multiple comparisons problem applied to attention analysis. The appropriate response is to define your hypothesis before examining the attention matrices, then test it on held-out sentences rather than confirming it on the same sentence you used to generate the hypothesis.
This worked example illustrates the general shape of attention visualization's epistemic situation: the visualizations are informative and hypothesis-generating, but confirming those hypotheses requires the kind of controlled analysis covered in subsequent chapters. The visual output is the beginning of the investigation, not its conclusion.