Weight Tying › Why Weight Tying Makes Sense

Weight tying reduces memory use, but its theoretical justification also explains why input and output embeddings should be related. Understanding this justification deepens your intuition for how transformers represent language.

Consider what each embedding encodes over the course of training. The input embedding for token $t$ gets updated whenever token $t$ appears in the input, with gradients flowing backward from the loss through all the transformer layers above. It learns a representation that makes downstream processing easier: similar tokens develop similar embeddings, tokens that play similar grammatical roles cluster together, and the geometric structure of the embedding space reflects the distributional statistics of the language.

The output embedding for token $t$ gets updated whenever token $t$ should be predicted as the next word, with gradients flowing directly from the cross-entropy loss. It learns what kind of hidden state the transformer should produce when "token $t$ is the right answer." The hidden state that generates "cat" should look similar to the output embedding for "cat."

Both representations capture the same fundamental thing: the contextual signature of token $t$ within the model's learned world model. A token's input representation and its output representation are both shaped by the same distributional statistics of the language. They are really two approximations to the same underlying concept, learned through slightly different optimization paths.

Here is a simple argument for why tying should help. Suppose you have two separate matrices, $\mathbf{E}$ and $\mathbf{W}_{\text{out}}$, and the model learns them independently. After training, you look at the embedding for "dog" in the input matrix and the embedding for "dog" in the output matrix. If the model has learned well, these two vectors should be similar, because both capture the meaning of "dog." But they are not required to be similar, so the model wastes capacity representing the same concept twice in slightly different ways.

Weight tying removes this redundancy by decree. The model is forced to find a single representation that works for both purposes, which means every gradient update contributes to a more coherent and consistent token representation. You are effectively doubling the amount of learning signal that shapes each token's embedding, because both the input path and the output path update the same weights.

This consistency argument applies directly to language modeling, where the input and output vocabularies are identical. The model reads and writes in the same language, using the same tokens. There is no fundamental reason why the concept of "dog" should be represented differently depending on whether the model is reading it or generating it.

Research has consistently confirmed that weight tying helps rather than hurts performance:

• Press and Wolf (2017) introduced the technique systematically and demonstrated that tying input and output embeddings improves perplexity across language modeling benchmarks, despite having fewer parameters. This was the result that established weight tying as a standard practice.

• Inan et al. (2017) provided complementary theoretical analysis, connecting weight tying to a regularization perspective where the shared matrix is encouraged to satisfy both input and output objectives simultaneously.

• Practical adoption: Most modern language models, including GPT-2, BERT, RoBERTa, T5, and their descendants, use weight tying by default. The technique has become so standard that its use is often not even mentioned in architecture descriptions.

The improvement in perplexity also reflects more than the regularization from having fewer parameters. Tied embeddings tend to learn better representations because gradients from the output loss flow directly into the input embeddings, and gradients from the input embedding usage flow into the output predictions. The two optimization objectives reinforce each other.