LLaMA (2023) › The Solution

Meta's FAIR team addressed these challenges by designing and releasing LLaMA, a family of efficiently architected, openly accessible language models that combined optimal training strategies with proven architectural innovations. The solution involved multiple coordinated components: following compute-optimal training principles, implementing efficient architectural improvements, curating high-quality training data, and establishing an open release strategy that enabled broader research access while maintaining appropriate safeguards.

The foundation of LLaMA's design was the application of Chinchilla scaling laws, which had demonstrated that models trained with approximately 20 tokens per parameter could achieve optimal performance for a given computational budget. Rather than simply scaling up model size, the LLaMA team carefully balanced model parameters with training data to maximize efficiency. LLaMA-7B, the smallest model, was trained on 1 trillion tokens, while LLaMA-65B, the largest, was trained on 1.4 trillion tokens. This approach ensured that each model followed compute-optimal principles, achieving competitive performance despite having fewer parameters than models like GPT-3.

The training data for LLaMA was carefully curated from multiple high-quality sources, including Common Crawl, C4, GitHub, Wikipedia, books, and ArXiv papers. The data preprocessing pipeline applied several quality filters: removing low-quality content, deduplicating documents, and identifying and filtering inappropriate material. This curation process ensured that the models were trained on diverse, high-quality text that would enable strong performance across a wide range of tasks. The emphasis on data quality, combined with the optimal data-to-parameter ratio, enabled LLaMA models to achieve exceptional performance relative to their size.

LLaMA incorporated several architectural improvements that enhanced both training efficiency and model performance. The first innovation was the use of RMSNorm (Root Mean Square Layer Normalization) instead of standard LayerNorm. RMSNorm simplified the normalization computation by removing the mean-centering step, requiring only the calculation of the root mean square. This simplification reduced computational overhead while maintaining training stability, enabling more efficient training and inference.

The second key architectural change was the adoption of SwiGLU activation functions in the feed-forward networks. SwiGLU, which combines the Swish activation with Gated Linear Units, had been shown in prior research to improve model performance compared to standard ReLU activations. However, SwiGLU requires more parameters than ReLU because it includes a gating mechanism. To maintain parameter efficiency, LLaMA used a dimension of $\frac{8}{3}d$ for the SwiGLU layers instead of the standard $4d$ used in many transformer architectures, where $d$ is the model dimension. This adjustment compensated for the additional parameters while preserving the performance benefits of SwiGLU.

The third architectural innovation was the use of rotary positional embeddings (RoPE) instead of absolute positional embeddings. RoPE encodes positional information by rotating query and key vectors in a way that naturally incorporates relative position relationships. This approach had been shown to improve model performance on tasks requiring long-range dependencies and to enable better generalization to longer sequences than the training context length. For LLaMA, RoPE enabled more effective handling of positional information while maintaining computational efficiency.

The combination of compute-optimal training and architectural improvements enabled LLaMA to achieve exceptional training efficiency. The models were trained using standard transformer training techniques with AdamW optimization, cosine learning rate scheduling, and gradient clipping. The efficient architecture reduced per-iteration computational costs, while the optimal data-to-parameter ratio ensured that training compute was used effectively. This efficiency made it feasible to train multiple model sizes, enabling the release of a family of models that could serve different use cases and computational constraints.

The training infrastructure used Meta's computational resources, but the models themselves could be efficiently fine-tuned and deployed on more modest hardware. The smaller models in the LLaMA family, particularly LLaMA-7B and LLaMA-13B, could be run on single high-end GPUs, making them accessible for researchers and organizations with limited computational resources. This accessibility was a key goal of the project: enabling broader participation in language model research and development.

The release strategy for LLaMA balanced the goals of open research with responsible deployment considerations. Model weights were made available to researchers through an application process, with a license that allowed for research and commercial use subject to certain conditions. This approach enabled researchers to access model weights directly, conduct their own evaluations, fine-tune models for specific applications, and study model internals, while maintaining some oversight to ensure responsible use.

The open nature of LLaMA enabled immediate and extensive research activity. Researchers could download model weights, examine architectures in detail, run custom evaluations, and build applications without dependency on external APIs. This access made studies possible that proprietary models had prevented. Researchers could investigate model representations, conduct safety analyses, develop fine-tuning techniques, and explore applications that required model modifications or custom inference configurations.