ALBERT › Summary

ALBERT introduced two key innovations for building more efficient transformers:

• Factorized embeddings decompose the large embedding matrix into two smaller matrices. Instead of directly mapping vocabulary tokens to hidden-dimension vectors (requiring $V \times H$ parameters), ALBERT first maps to a smaller embedding space ($V \times E$ parameters) then projects up ($E \times H$ parameters). When the embedding dimension $E$ is much smaller than the hidden dimension $H$, this reduces embedding parameters by up to 80%.

• Cross-layer parameter sharing uses a single set of transformer weights applied repeatedly, cutting layer parameters by a factor equal to the number of layers.

• Sentence Order Prediction replaces the easier NSP task with order detection, requiring discourse understanding rather than topic matching.

• Iterative refinement emerges from shared parameters, as the model applies the same transformation multiple times to progressively refine representations.

• Memory benefits during training come from smaller optimizer states and gradient buffers, though inference speed matches BERT due to identical computation.

• Performance parity demonstrated that model quality depends more on computational depth and width than on unique parameter count, with ALBERT-xxLarge achieving state-of-the-art results.

ALBERT showed that careful architecture design could dramatically reduce model size without sacrificing performance, influencing subsequent work on efficient transformers.