Vision-Language Projection › Summary

Vision-language projection solves the basic architectural challenge of connecting two pre-trained models that were trained in isolation on entirely different modalities. The three main approaches represent a progression from simple to expressive:

Linear projection uses a single weight matrix to map each vision token from the vision encoder's dimension to the LLM's dimension. It is parameter-efficient (roughly 4 million parameters for a ViT-L/14 to LLaMA-2-7B bridge), preserves spatial structure faithfully, and optimizes cleanly. It works well when the vision encoder was trained contrastively and its representations are already partially aligned with language. Its principal limitation is that it preserves the full visual token sequence, which can exhaust context budgets for high-resolution or multi-image inputs.

MLP projection introduces non-linear transformations through hidden layers and activation functions, letting the projection to learn more complex feature re-organizations that pure linear transformations cannot represent. This proves valuable when vision and language encoders originate from very different training paradigms, or when tasks require fine-grained visual distinctions that need non-linear mapping into language space. The cost is increased parameter count and greater sensitivity to initialization and learning rate choices.

Q-Former addresses the token budget problem through attention-based compression. A set of fixed learnable query embeddings attend to the visual tokens via cross-attention, extracting a fixed-size summary regardless of how many input patches the vision encoder produces. The 6x-8x compression (from 197-257 tokens down to 32) frees substantial context window capacity, letting longer conversations and multi-image inputs at the cost of potential fine-grained detail loss and substantially higher training complexity.

Training strategies complement the architectural choices. Freezing both the vision encoder and LLM while training only the projection is the standard baseline: it preserves pre-trained capabilities, prevents catastrophic forgetting, and minimizes compute requirements. Stage-wise training, first aligning modalities on large noisy datasets and then fine-tuning for instruction-following on curated data, produces projections that both accurately translate visual concepts and effectively support conversational reasoning. Loss masking on visual token positions ensures the projection learns to produce representations useful for text generation, not to reconstruct input features.

Together, these projection mechanisms have made the current generation of multimodal AI systems practical, letting the composition of specialized pre-trained models into unified vision-language systems capable of reasoning about visual content without requiring end-to-end training from scratch.