LLaVA Architecture › LLaVA Architecture

Large Language and Vision Assistant (LLaVA) changes how we approach multimodal AI. Before LLaVA, the dominant approach for creating vision-language systems involved training large models from scratch on billion-scale image-text datasets, an approach that demanded extraordinary computational resources and often resulted in models that struggled to match the reasoning capabilities of pure language models. Rather than training large vision-language models from scratch, LLaVA shows that we can repurpose useful pre-trained components, a frozen vision encoder and a frozen large language model, connecting them through a surprisingly simple architectural bridge. This approach, known as visual instruction tuning, turns a text-only assistant into a multimodal conversationalist able to understand images, reason about visual content, and follow complex instructions that span both modalities.

The insight driving LLaVA is elegant in its efficiency. Computer vision has produced exceptional encoders like CLIP's Vision Transformer, which capture rich visual representations through extensive pre-training on hundreds of millions of image-text pairs. These encoders have learned to map visual content into rich, semantically real vector spaces where similar concepts cluster together. Separately, large language models like LLaMA and Vicuna have developed remarkable reasoning and instruction-following capabilities through extensive text-only training on trillions of tokens. These models have acquired world knowledge, logical reasoning abilities, and the capacity to follow complex instructions through next-token prediction. Rather than training a new model to acquire both skill sets simultaneously, which would require learning visual perception and linguistic reasoning from scratch, LLaVA asks a simpler question: can we teach these existing models to communicate with each other?

The answer is yes, through a lightweight projection layer that translates visual features into the linguistic embedding space that the language model understands. This chapter explores how LLaVA constructs this bridge, the two-stage training strategy that makes it possible, how visual instruction data is generated at scale, and how the architecture has evolved through subsequent iterations to handle higher resolutions and more complex visual reasoning tasks.