Logit Lens
Language AI HandbookAll chapters Logit LensSection view · Sections in this chapter1 / 12 ↓
Reading tools Definitions, links & notes
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When a transformer processes text, it runs each token representation through dozens of layers before producing the final output. For most of machine learning's history, those intermediate layers were treated as black boxes, their representations inscrutable. The logit lens changes that. It asks a simple but powerful question: if we stopped the model mid-computation and asked it "what word do you predict right now?", what would it say?
The answer turns out to be deeply informative. By projecting intermediate hidden states directly into vocabulary space, the logit lens reveals how a model's prediction evolves layer by layer, how early layers are confused or wrong, and how later layers converge on the final answer. This makes it one of the most direct tools for understanding what transformers do as they process language.
As we saw in the Attention Visualization and Probing Classifiers chapters, understanding what information is encoded at each layer is central to transformer interpretability. The logit lens offers a complementary perspective: rather than asking what a linear probe can classify, it directly reads out the model's implicit prediction at each layer. You install no extra machinery, train nothing, and make no architectural modifications. The logit lens is purely an analytical technique that repurposes the model's existing output machinery to peer inside its processing.
Comments
Add to the conversation
A question, a correction, or an example from your own work.
No comments yet. Be the first to share your thoughts!