Vision-Language Projection › Key Parameters

The key parameters for vision-language projection control the basic capacity and efficiency tradeoffs of the cross-modal bridge:

• vision_dim: The dimension of the vision encoder output (e.g., 1024 for ViT-L/14). This determines the input size to the projection layer. Larger vision models like ViT-H/14 (1280 dimensions) or ViT-G (1664 dimensions) provide richer visual features but require larger projection matrices. The choice of vision encoder is often the most consequential design decision for the quality of visual understanding.

• llm_dim: The dimension of the language model embedding space (e.g., 4096 for LLaMA-2-7B, 8192 for LLaMA-70B). The projection output must match this dimension exactly. Larger language models typically have larger embedding dimensions, requiring more projection parameters. The projection must learn to fill a higher-dimensional space with meaningful visual information, which generally benefits from more expressive projection architectures.

• bias: Whether to include bias terms in linear projections (typically False in modern implementations). When the projection is followed by Layer Normalization (which is common), the bias term is redundant since LayerNorm has its own learnable shift parameter. Omitting the bias reduces the parameter count slightly and can improve optimization stability.

• hidden_dim: For MLP projections, the intermediate dimension (often $4 \times$ vision_dim) that increases representational capacity. The expansion ratio of four mirrors the standard transformer feed-forward network design. Increasing this ratio allows more complex non-linear transformations but increases both parameter count and the risk of overfitting.

• num_queries: For Q-Former, the number of learnable query tokens (typically 32) that compress variable-length visual features into a fixed sequence. Increasing this number preserves more visual detail and improves performance on tasks requiring fine-grained spatial understanding, but reduces the compression benefit and increases the visual token budget consumed from the LLM's context window.

• qformer_dim: The hidden dimension for Q-Former transformer layers (e.g., 768, matching BERT-base). This controls the expressiveness of the cross-attention mechanism and the information capacity of each query token. The BLIP-2 authors used 768 to reuse BERT-base weights for initialization, accelerating convergence.

• qformer_heads: The number of attention heads in Q-Former layers, letting parallel extraction of different aspects of visual information. Standard configurations use 12 heads matching BERT-base, though the optimal count depends on the dimensionality and the diversity of visual information that must be captured.

• qformer_layers: The depth of the Q-Former transformer, controlling the iterative refinement of query representations. More layers allow queries to progressively integrate information across the image, with early layers capturing local features and later layers building global context, at the cost of additional computation and training time.