LLaVA Architecture › Why Simple Connectors Work

Before diving into the architecture, it is worth understanding why such a simple connection approach succeeds at all. The short answer is that both CLIP and large language models have already done the hardest work independently.

CLIP's training objective is explicitly cross-modal: it learns to align images and text descriptions in a shared embedding space using contrastive learning. An image of a cat and the text "a photograph of a cat" are pulled toward the same region of embedding space, while the image of a cat and the text "an elephant in water" are pushed apart. After training on 400 million image-text pairs, CLIP's visual features are not raw pixel statistics but semantic concepts, organized in a way that shows linguistic categories. The concept of "dog" in CLIP's visual space is already related to "canine," "pet," and "animal" in a way that mirrors how these words cluster in the embedding space of a language model.

This semantic overlap between CLIP's visual space and the language model's text space is what makes a simple linear projection viable. The projection layer does not need to teach the language model what a dog is. The LLM already knows. The projection layer only needs to translate the CLIP representation of a dog into the specific numerical coordinates the LLM uses internally for dog-related concepts. This is a much easier problem than building a bridge between entirely foreign spaces.

The language model contributes a complementary capability. Through instruction fine-tuning (covered in depth in earlier chapters), models like Vicuna have learned to follow instructions expressed as natural language. They understand "describe the image," "explain what is happening," and "what is the person on the left holding?" as requests requiring specific response patterns. What they lack is the perceptual input to ground these instructions in actual visual content. LLaVA supplies exactly that: a stream of semantically real visual tokens that the LLM can treat as additional context, much like a passage of text that describes the image.

The reason a linear projection suffices for initial alignment, and a two-layer MLP suffices for fine-grained alignment, is that the two representation spaces are already structurally compatible thanks to CLIP's contrastive training. This compatibility is the key empirical finding that LLaVA shows, and it has reshaped how the field thinks about multimodal architecture.

You can get a concrete sense of this compatibility by looking at nearest-neighbor relationships. Take any visual feature vector from CLIP's encoder for an image of, say, a sunset over mountains. Compute the cosine similarity between that vector and the word embedding vectors for hundreds of English words in a language model. You will find that "sunset," "horizon," "dusk," "mountains," and "orange" rank among the nearest neighbors, even though the CLIP encoder and the LLM were trained separately on different data with different objectives. This cross-model semantic coherence is not a coincidence. Both models were trained on internet data that shows the same underlying associations between visual and linguistic concepts in human culture.

The projection layer exploits this pre-existing coherence by learning a relatively simple transformation that aligns the scales and orientations of the two vector spaces. Researchers have verified this experimentally: after projecting CLIP features through a random (untrained) linear layer, the projected features still contain useful visual information. A trained projection layer simply optimizes this alignment more precisely, improving the LLM's ability to read the visual tokens accurately.

This also explains why LLaVA's approach generalizes across different LLM architectures. Whether you use LLaMA, Mistral, Phi, or Qwen as the language model backbone, the same CLIP vision encoder can be connected through a trained MLP and reach strong multimodal performance. The universal cross-architecture compatibility points to the underlying semantic structure being a property of internet-scale training data rather than any specific architectural choice.