LLaMA Architecture › Limitations

LLaMA's achievements should be understood in context of its limitations, some of which shaped the evolution of subsequent models.

The original LLaMA models were pretrained language models, not instruction-following assistants. A raw LLaMA model responds to prompts by continuing text in the style of its training data rather than following user instructions in a helpful, conversational way. This limitation was widely recognized: the early community derivatives like Alpaca and Vicuna existed specifically to address it by fine-tuning LLaMA on instruction-response pairs. The pretrained weights were a foundation, not a finished product.

The 2048-token context window was a significant practical constraint. Many real-world applications, including document summarization, long-form question answering, and code understanding, require much longer contexts. Subsequent models addressed this through RoPE interpolation and extended pretraining on long documents, but the original LLaMA was limited to roughly one to two pages of text at a time.

The training data, despite aggressive filtering, remained predominantly English. Non-English text appeared only in portions of CommonCrawl and Wikipedia. As a result, LLaMA's performance on non-English languages lagged significantly behind its English capabilities, which makes it a less useful foundation for multilingual applications. Models like Qwen and Yi, which included Chinese-language data as a first-class training priority, emerged partly to fill this gap.

LLaMA also reflected the biases present in its training data. Web text and books from the early 2020s carry the same social biases, factual errors, and cultural assumptions embedded in that text. The perplexity filtering removed low-quality writing but did not specifically target harmful stereotypes, misinformation, or sensitive content. Instruction fine-tuning and RLHF applied by derivative models provided some mitigation, but the base model's biases remained a concern for direct deployment.

Finally, the 2048-token context constraint interacted poorly with the tokenization choice to split numbers into individual digits. A moderately long mathematical document could consume a disproportionate number of tokens, reducing effective context length further. This was a known limitation that later models addressed partly through expanded vocabulary sizes and context windows.