Vision-Language Projection › Worked Example: Dimensional Transformation

Consider building a vision-language model using CLIP ViT-L/14 as the vision encoder and LLaMA-2-7B as the language model. This combination represents a common architectural choice in open-source vision-language models, combining the strong zero-shot visual capabilities of CLIP with the reliable reasoning abilities of LLaMA.

Vision Encoder Output:

The ViT-L/14 processes a 224×224 pixel image by dividing it into 14×14 patches. The number of patch tokens is $(224/14)^2 = 256$. Adding the class token yields 257 visual tokens in total. Each token is a 1024-dimensional vector, giving a vision encoder output tensor of shape $[257, 1024]$.

The choice of ViT-L/14 provides a good balance between computational efficiency and representational capacity. The 14×14 patch size offers finer spatial resolution than the more common 16×16, capturing more detail in the visual features, while the 1024-dimensional embeddings provide rich representations for each patch.

Target LLM Input:

LLaMA-2-7B uses a 4096-dimensional embedding space. The projection must transform each 1024-dimensional vision vector into a 4096-dimensional LLM vector, an expansion by a factor of four.

Linear Projection:

The weight matrix has shape $1024 \times 4096$, containing $4{,}194{,}304$ parameters. For each of the 257 visual tokens, we compute:

$$\mathbf{h}_i = \mathbf{v}_i \mathbf{W}, \quad \mathbf{v}_i \in \mathbb{R}^{1024}, \quad \mathbf{W} \in \mathbb{R}^{1024 \times 4096}, \quad \mathbf{h}_i \in \mathbb{R}^{4096}$$

The output tensor has shape $[257, 4096]$: 257 visual tokens, each now in LLaMA's 4096-dimensional space. These 257 tokens are then prepended to the text token embeddings, and the full sequence is fed into LLaMA-2-7B for generation. The 257 visual tokens represent 6.3% of LLaMA's 4096-token context window, leaving 3839 tokens for text.

The linear projection adds only 4.2 million parameters, a tiny fraction of the billions in the LLM. Despite its simplicity, this projection must perform the important mapping from CLIP's contrastive visual space to LLaMA's autoregressive linguistic space.

Q-Former Compression:

The Q-Former uses 32 learnable queries, each of dimension 768. The cross-attention compresses the 257 input tokens down to 32 output tokens of dimension 768. A final linear projection maps these 32 vectors from 768 dimensions to 4096 dimensions (LLaMA's space), adding $768 \times 4096 = 3{,}145{,}728$ additional parameters.

Using Q-Former, 257 visual tokens that would occupy 6.3% of the context window become only 32 tokens, leaving 4064 positions for text. Processing two images with linear projection would require 514 tokens (12.5% of the context), while Q-Former would require only 64 tokens (1.6%). For multi-image or multi-turn visual conversations, this difference in context efficiency is decisive.

The tradeoff is information compression. The Q-Former must decide which of the 257 input patches' information to retain in its 32 output tokens. For simple image captioning, 32 tokens may be sufficient to capture the key objects and scene context. For tasks requiring precise localization or counting, the compression may lose fine-grained spatial detail that the language model would have needed.