LLaVA Architecture › LLaVA-NeXT (1.6): Dynamic High Resolution

Building on the grid approach of 1.5, LLaVA-NeXT (also called LLaVA 1.6) introduces dynamic high-resolution processing and improved visual reasoning capabilities. This iteration addresses one of the basic limitations of fixed-resolution vision encoders: the inability to adapt to varying image sizes and aspect ratios while maintaining computational efficiency.

Rather than resizing all images to a fixed square, which distorts aspect ratios and can obscure important details in non-square images, LLaVA-NeXT maintains aspect ratios and processes images at their native resolution (up to a maximum). The strategy involves several key mechanisms:

Aspect Ratio Preservation: Images are divided into grids that match predefined aspect ratio buckets (1:1, 1:2, 2:1, 1:3, 3:1, and so on). Rather than stretching a wide panoramic image into a square, the system selects the grid setup whose total shape most closely matches the image's native aspect ratio, then pads only minimally to fill the chosen grid. This bucketing system ensures that images are processed without the geometric distortion caused by naive resizing to squares. A person photographed in portrait orientation is processed in a tall grid, not a distorted square.

Variable Token Length: Different images produce different numbers of visual tokens based on their resolution and aspect ratio. A panoramic image might generate twice as many tokens as a square image of similar total area, letting the model to process wide scenes without losing detail on either the left or right extremes. This variability requires the LLM to handle sequences of differing lengths, which modern transformer architectures accommodate through dynamic attention masking.

Global Context Thumbnail: Alongside the tiled high-resolution crops, LLaVA-NeXT always includes a downsampled version of the full image as a global context thumbnail, processed through the same vision encoder at standard resolution. This global thumbnail gives the model an overall view of the scene even when the tiled crops are individually small excerpts. Without this thumbnail, the model examining individual tiles might lose track of the overall spatial composition, mistaking a foreground object tile for the entire scene. The thumbnail ensures that the model can answer questions like "what is in the background of this image" even when the high-resolution processing partitions the image into small patches.

This approach allows the model to read small text in images, recognize fine details in complex scenes, and handle documents and charts more effectively than fixed-resolution approaches. The combination of global context and local detail is particularly powerful for document understanding tasks where a user might ask "summarize this page" (requiring global understanding) and "what does the footnote say" (requiring local high-resolution detail) in the same conversation.

LLaVA-NeXT employs more advanced data mixing strategies, balancing multiple types of visual understanding:

• General visual conversation for everyday scene understanding
• OCR and document understanding for text-rich images
• Multi-language visual data to support non-English queries
• Scientific diagrams and charts for specialized domain knowledge

The training recipe also extends to different base LLMs, including Mistral and other modern architectures beyond the original Vicuna. This shows the generalizability of the LLaVA approach. By varying the base language model, researchers can confirm that the visual instruction tuning methodology is reliable across diverse underlying language architectures, and that the key capability resides in the training pipeline rather than in any particular LLM's architecture.