Open LLM Wave (2023) › The Solution

The open model wave of 2023 addressed these challenges through a coordinated effort by multiple organizations to develop and release high-quality open language models. Each organization brought different technical approaches, architectural innovations, and training strategies, creating a diverse ecosystem that offered alternatives to proprietary systems while enabling new use cases and research directions.

MosaicML's MPT (Mosaic Pretrained Transformer) models, released starting in May 2023, demonstrated that open models could achieve competitive performance through careful architectural design and training efficiency. The MPT models introduced innovations such as ALiBi (Attention with Linear Biases) for better handling of variable context lengths, and they were trained on carefully curated datasets including code, mathematical text, and web content. MPT-7B and MPT-30B provided strong performance while remaining accessible for organizations with modest computational resources, and MosaicML released the models under permissive Apache 2.0 licensing that enabled commercial use and modification.

The Technology Innovation Institute's Falcon models, released in June 2023, took a different approach to achieving competitive performance. Falcon-40B and Falcon-7B were trained on a massive, carefully filtered dataset called RefinedWeb, which contained trillions of tokens of high-quality web text. The Falcon models emphasized the value of data quality over quantity. This showed that with better data curation, smaller models could achieve performance comparable to larger systems trained on less carefully filtered datasets. Falcon models also featured efficient architectures optimized for inference speed and memory usage, making them practical for deployment in resource-constrained environments.

Mistral AI's Mistral models, starting with Mistral 7B in September 2023, represented another significant contribution to the open ecosystem. Mistral 7B demonstrated that careful architectural choices and training techniques could produce models that outperformed much larger proprietary systems on certain benchmarks. The model used innovations such as grouped-query attention and sliding window attention to improve efficiency, while also showing strong performance on reasoning tasks and multilingual capabilities. Mistral AI's approach emphasized both model quality and deployment practicality, with models designed to run efficiently on consumer hardware.

Beyond these major releases, 2023 saw numerous other open models that explored different aspects of language model development. Organizations experimented with various architectures, training data compositions, optimization techniques, and licensing approaches. This diversity created a rich ecosystem where developers and researchers could choose models that best fit their specific needs, whether prioritizing performance, efficiency, licensing flexibility, or specific capabilities like code generation or multilingual support.

The open models of 2023 also led to complementary tooling and infrastructure. As models became available for local deployment, the ecosystem responded with improved frameworks for model serving, quantization techniques for reducing memory requirements, and fine-tuning libraries optimized for open models. Projects like Hugging Face's Transformers library expanded support for these new models, while communities developed evaluation frameworks, deployment guides, and optimization techniques specific to open model deployment.

The licensing choices of these open models also addressed different use cases and requirements. Some models, like MPT, used permissive Apache 2.0 licenses that enabled unrestricted commercial use. Others, like LLaMA-based models, used licenses that allowed commercial use but required adherence to acceptable use policies. This variety in licensing gave organizations flexibility to choose models that aligned with their legal requirements and business needs.