LLaVA Architecture › LLaVA 1.5: Architectural Refinements

LLaVA 1.5 introduced several necessary improvements that materially boosted performance without fundamentally changing the architecture. These refinements show how careful engineering choices in data and projection design matter as much as the core concept. By iterating on the connector design and training data mixture, the authors showed that the simple LLaVA framework had substantial headroom for improvement without requiring architectural complexity.

The original LLaVA used a simple linear projection from vision to language space. While effective, this linear mapping assumes a relatively simple relationship between visual and linguistic representations. LLaVA 1.5 replaces this with a two-layer MLP with GELU activation. This non-linear mapping transforms visual features through an intermediate representation before projecting to the LLM space:

$$\begin{aligned} H_{\text{mid}} &= \text{GELU}(W_1 Z + b_1) \\ H_v &= W_2 H_{\text{mid}} + b_2 \end{aligned}$$

where:

• $Z$: the visual feature vectors from the frozen vision encoder (shape $N \times d_v$)
• $W_1, b_1$: the weight matrix and bias for the first linear layer (mapping from $d_v$ to $d_{\text{LLM}}$)
• $\text{GELU}$: the Gaussian Error Linear Unit activation function, introducing non-linearity between layers
• $H_{\text{mid}}$: the intermediate representation after the first linear layer and activation
• $W_2, b_2$: the weight matrix and bias for the second linear layer (mapping within $d_{\text{LLM}}$)
• $H_v$: the final projected visual embeddings ready for input to the LLM

Why does the MLP outperform the linear projection? Consider what the linear layer can and cannot stand for. A linear map can rotate and scale while shearing the visual feature space, but it cannot introduce curvature or stand for non-linear relationships between visual concepts. For instance, if "red" and "round" are both present in the visual features, a linear layer can adjust their magnitudes but cannot compute "red AND round implies apple" in any way that departs from a linear combination. The intermediate hidden layer and GELU activation allow the network to compose visual features in non-linear ways, capturing richer semantic structure before projecting to the LLM space.

This non-linear connector gives greater capacity to map complex visual concepts to linguistic embeddings. The added parameters (though still small relative to the LLM, typically less than 1% of total model parameters) allow for more fine-grained translation of visual features, particularly for abstract concepts and fine-grained visual details that might not align linearly with text embeddings.

LLaVA 1.5 incorporates additional training data from academic visual question answering datasets (VQA-v2, GQA, OKVQA, OCR-VQA) during the second stage. These datasets give high-quality, human-annotated question-answer pairs covering diverse reasoning types including compositional questions, reading comprehension, and knowledge-based reasoning.

However, rather than using the datasets' original formats, which often include dataset-specific biases or formatting artifacts, LLaVA 1.5 reformats them into simple, consistent prompts. Instead of complex VQA templates like "What is the color of the object left of the person?", the data uses direct questions: "What color is the car?" This approach maintains the diversity of academic datasets while preventing the model from overfitting to specific dataset formats or answer distributions. It also shows that visual instruction tuning can subsume traditional visual QA benchmarks when properly formatted. By stripping away dataset-specific formatting, the model learns the underlying reasoning capabilities rather than memorizing stylistic patterns associated with particular benchmarks.

The combination of synthetic GPT-generated data and curated academic benchmarks creates a training mixture that is both diverse (from the synthetic data) and reliable (from human-verified academic annotations). This dual-source strategy has become standard practice in modern multimodal training pipelines, and the balance between synthetic and human-verified data remains an active research question.

Original LLaVA used 224px or 336px image resolution, which limits the model's ability to perceive fine details such as small text, distant faces, or detailed textures. LLaVA 1.5 supports higher resolutions by dividing images into grids and processing each crop separately through the vision encoder.

The tiling approach works by dividing a high-resolution image into non-overlapping crops, each of size matching the vision encoder's native input resolution (336px). A 672px image yields a 2x2 grid of four 336px crops, each processed independently. The visual tokens from all crops are concatenated and projected together, letting the model to examine image regions at higher fidelity without requiring a modified vision encoder. For example, if a document image contains small text that falls within a single 336px crop, the model can read it by examining that crop's tokens at full resolution, even though the full document would be too large for the encoder to process at once.

This tiling approach is not without cost. A 2x2 grid quadruples the number of visual tokens from 576 to 2304, a 3x3 grid produces 5184 tokens, and each token adds to the sequence length that the LLM must process through its quadratic attention mechanism. Longer sequences mean quadratically increasing memory and computation at inference time, creating a direct tradeoff between visual resolution and processing cost that scales unfavorably for very high-resolution images.