LLaMA (2023) › The Problem

Large language model research in early 2023 faced an access problem that limited the scope and pace of innovation. State-of-the-art models like GPT-3, PaLM, and other proprietary systems were only accessible through API interfaces provided by their creators. While these APIs enabled many applications and use cases, they created significant limitations for researchers and developers who wanted to study, modify, fine-tune, or build upon these models. The black-box nature of API access meant researchers could not examine internal model representations, experiment with architectural modifications, conduct detailed safety analyses, or fine-tune models for specialized domains without relying on external services.

Academic researchers faced particular challenges in this environment. Conducting research on language models required either securing expensive API access with usage limits and restrictions, or training models from scratch, which demanded computational resources that most academic institutions could not afford. The gap between what proprietary systems could do and what researchers could study or build was enormous, creating a situation where most of the field's innovation was concentrated in a few well-resourced organizations. This concentration limited the diversity of perspectives, use cases, and safety research that could emerge from broader participation in language model development.

The proprietary models themselves, while impressive, had not necessarily been designed with optimal efficiency in mind. GPT-3, released in 2020, had 175 billion parameters and was trained on approximately 300 billion tokens, representing a ratio of roughly 1.7 tokens per parameter. However, research published in 2022, particularly the Chinchilla scaling laws, had demonstrated that this approach was suboptimal. Models with far fewer parameters could achieve competitive or superior performance if trained on substantially more data, following a compute-optimal ratio of approximately 20 tokens per parameter. This inefficiency meant that the proprietary models were inaccessible and potentially suboptimal in their use of computational resources.

The architecture of existing models, while effective, had room for improvement. Standard transformer architectures used LayerNorm, ReLU activations, and absolute positional embeddings, all of which had been the subject of research showing that alternative approaches could improve training stability, model performance, or both. However, implementing these improvements required access to model weights and the ability to modify architectures, which proprietary models did not allow. Researchers who wanted to explore architectural improvements had to either work with much smaller models that did not reflect the scale of production systems, or accept the limitations of API-based access to larger models.

The evaluation and benchmarking of language models also suffered from the access problem. Researchers could not run custom evaluations on proprietary models beyond what the API interfaces allowed, limiting the types of analyses that could be conducted. Questions about model internals, training data composition, and detailed performance characteristics remained opaque, making it difficult to understand how these models worked, what their limitations were, or how they could be improved. The research community needed models that researchers could examine and modify, including through custom evaluations.

The cost structure of using proprietary models created barriers for many potential applications. While API access enabled many use cases, applications requiring extensive fine-tuning, custom inference configurations, or offline deployment could not rely on external APIs. Organizations that wanted to deploy language models in production with specific latency requirements, privacy constraints, or cost considerations found API-based access inadequate. The field needed open models that could be deployed independently, fine-tuned for specific domains, and optimized for particular use cases without dependency on external services.

Finally, the safety and ethics research community faced particular challenges. Understanding model biases, failure modes, and safety properties required deep access to model internals and the ability to conduct extensive testing. The black-box nature of proprietary models limited the types of safety research that could be conducted, making it harder to identify and address potential harms. The field needed open models that researchers could study thoroughly, enabling deeper safety analysis and the development of better safety techniques.