LLaVA Architecture › Key Architectural Parameters

The key architectural parameters for LLaVA determine the model's capacity, resolution capabilities, and computational requirements. Understanding these parameters helps calibrate expectations when adapting LLaVA for different applications or when choosing between model scales.

• vision_hidden_size: Dimension of the vision encoder's output features (1024 for CLIP ViT-L/14). This determines the size of visual representations before projection. Higher dimensions capture more visual detail but require larger projection matrices. Smaller CLIP variants use 768-dimensional features, while larger models can use up to 1280 dimensions.
• llm_hidden_size: Dimension of the language model's embedding space (4096 for LLaMA-7B). The projection layer maps vision features to this dimension. This is typically the largest dimension in the system and determines the width of all subsequent transformer layers. LLaMA-13B uses 5120 dimensions, and larger models scale further.
• num_patches: Number of visual tokens extracted from the image (576 for 336px resolution with 14px patches). This controls the spatial resolution of visual input. The formula is $(H / P) \times (W / P)$ where $H$ and $W$ are the image height and width and $P$ is the patch size. Higher resolution images processed with tiling can generate thousands of tokens.
• MLP connector: A two-layer projection network with GELU activation that transforms vision features to language space. The intermediate dimension typically matches $d_{\text{LLM}}$ (4096), maintaining constant width through the projection and giving the network enough capacity for non-linear alignment.

The number of visual tokens per image has a particularly strong influence on inference latency. With 576 tokens for a single 336px image, adding an image to the context is equivalent to adding a paragraph of text in terms of sequence length. For a 2x2 tiled image at 672px resolution, this becomes 2304 tokens, comparable to a full page of text. This has direct implications for user-facing applications where inference speed matters: models serving high-resolution image queries face a materially higher compute burden than those handling text-only queries or lower-resolution images.