V-JEPA 2 (2025) › The Problem: Limitations of Language-Centric AI

The remarkable success of large language models had obscured a fundamental limitation: their training was almost entirely text-based, which meant they learned about the world through linguistic descriptions rather than direct observation. While text contains vast amounts of information about the world, it represents reality through a particular lens, one that emphasizes verbal descriptions, narrative structures, and linguistic patterns rather than spatial relationships and the causal changes that unfold over time in physical interactions.

This limitation became particularly apparent when language models were asked to reason about physical scenarios. They could answer questions about physics concepts based on text descriptions, but they struggled with tasks that required understanding of spatial relationships, object properties, or cause-and-effect dynamics in concrete visual contexts. For example, a language model might correctly state that "gravity causes objects to fall," but it couldn't visually predict what would happen if a ball were dropped in a specific scene, or understand how the ball's trajectory would differ based on its position relative to other objects.

For applications requiring interaction with physical environments, this limitation was severe. Robotic systems needed to understand how actions would affect their surroundings, autonomous vehicles needed to predict how objects in a scene would move, and augmented reality systems needed to understand spatial relationships between virtual and physical objects. Text-based training provided insufficient foundation for these capabilities, as they required learning directly from visual observations and the consequences of actions in physical space.

The challenge was to develop architectures that could learn about the physical world through visual prediction and interaction, rather than through textual descriptions. This required fundamentally different approaches from those that had proven successful for language modeling. Visual data has different structure than text: it's continuous rather than discrete, spatial rather than sequential, and encodes information about physics and causality in ways that text descriptions cannot fully capture. Building AI systems that could learn from visual data to develop world models required new architectures, training paradigms, and learning objectives.