Open LLM Wave (2023) › Architectural Innovations

The open models of 2023 introduced several architectural innovations that improved efficiency and performance. These innovations demonstrated that thoughtful design choices could enable smaller models to achieve capabilities previously associated with much larger systems.

ALiBi (Attention with Linear Biases), used in MPT models, addressed the challenge of handling variable-length sequences by replacing fixed positional embeddings with learned linear biases that scaled with distance. This approach allowed models to generalize to longer sequences than those seen during training, an important capability for applications requiring long context windows. ALiBi's relative positioning approach also proved more efficient than traditional absolute positional encodings while providing better generalization.

Grouped-query attention, used in models like Mistral 7B, improved inference efficiency by reducing the number of key-value cache entries needed during generation. Instead of maintaining separate key-value pairs for each attention head, grouped-query attention shared key-value pairs across multiple query heads, significantly reducing memory requirements during inference while maintaining model quality. This optimization became particularly important as models were deployed in production environments with memory constraints.

Sliding window attention mechanisms allowed models to efficiently process very long sequences by limiting attention to a local window around each token while maintaining some global context through multiple attention layers. This approach enabled models to handle context lengths of 32,000 tokens or more while keeping computational costs manageable, opening new use cases that required processing long documents or maintaining extended conversation history.

The open models also explored different architectural choices for balancing model capacity with computational efficiency. Some models prioritized parameter count and model depth, while others focused on architectural efficiency and optimized training procedures. This diversity in approaches helped the community understand which design choices mattered most for different use cases and performance requirements.