LLaVA Architecture › Summary

LLaVA shows that multimodal intelligence emerges not from training large models from scratch, but from teaching existing expert models to communicate. The architecture's elegance lies in its restraint: by freezing both the vision encoder and language model during initial training, LLaVA focuses learning entirely on the translation layer between modalities, achieving remarkable capabilities with minimal trainable parameters.

Key insights from the LLaVA approach include:

• Modular alignment: Pre-trained vision encoders and language models can be connected through lightweight trainable projections, preserving the capabilities of both while letting cross-modal reasoning. This compositional approach allows each component to be updated independently as better models become available.

• Curriculum training: Separating feature alignment (Stage 1) from instruction tuning (Stage 2) creates stable training dynamics and prevents catastrophic forgetting of pre-trained capabilities. This staged approach mirrors human learning, where basic perception develops before complex linguistic reasoning.

• Visual instruction tuning: Extending instruction tuning to the multimodal domain requires carefully masked loss computation and diverse instruction data, which can be effectively generated using existing language models. This data-centric approach lets rapid scaling of training sets without large human annotation efforts.

• Architectural evolution: Simple changes, upgrading from linear to MLP projections and supporting dynamic resolutions, yield large improvements without abandoning the core approach. This suggests that the basic insight of modality alignment via projection is reliable and extensible.

• Emergent multimodal reasoning: Because visual tokens are processed by the same transformer attention mechanism as text tokens, the LLM's existing reasoning capabilities (logical inference, multi-step problem solving, commonsense reasoning) naturally extend to visual inputs without any specialized training on reasoning with images.

As we progress to the Flamingo architecture in the next chapter, we'll examine alternative approaches that sacrifice some of LLaVA's simplicity for greater expressiveness in cross-modal interactions, introducing mechanisms for dynamic attention between vision and language that LLaVA's frozen encoder design deliberately avoids.