Weight Tying › Weight Tying: Shared Embeddings in Transformers

Language models contain two large embedding matrices that seem to serve completely different purposes. One converts input tokens into continuous vectors at the start of the network. The other converts output vectors back into token probabilities at the end. If you examine these matrices for a model like GPT-2, you will find that each has 50,257 rows, one per vocabulary token, and 768 columns, one per hidden dimension. Together, they account for roughly 77 million parameters in a model whose total size is only about 117 million parameters. That means over 65 percent of all parameters in GPT-2 Small live in just these two matrices.

What makes this extraordinary is that both matrices answer fundamentally the same question: what does each vocabulary token mean within this model's learned representation space? The input embedding asks, "given this token, what vector should I pass into the transformer?" The output projection asks, "given this hidden state, how strongly does it correspond to each token?" These are two sides of the same coin. A model that has learned a rich representation for the word "justice" should encode it consistently whether reading it or generating it.

Weight tying exploits this insight with striking simplicity: instead of learning two separate matrices, the model uses one. The same embedding matrix that converts "justice" into a vector at the input also is the output target when the model wants to generate "justice". This single constraint halves the embedding-related parameters and, counter-intuitively, often improves model quality. You might expect that giving the model fewer degrees of freedom would hurt performance, but the opposite tends to be true. The constraint acts as a useful regularizer that forces coherent representations.

This chapter covers weight tying from its mathematical foundations to its practical implementation. We will understand exactly why tying makes theoretical sense, trace through numerical examples to see how shared embeddings work in practice, examine the gradient dynamics that make tied models learn differently, and explore encoder-decoder extensions where three matrices can be collapsed into one. We will also study the limitations: tying is not always appropriate, and knowing when to avoid it is as important as knowing how to implement it.

By the end of this chapter, you will understand one of the most universally adopted architectural choices in modern language modeling, appearing in GPT-2, BERT, T5, and nearly every transformer trained since 2017.