Multimodal Applications › Video Understanding

Video introduces the temporal dimension that static images lack. A single frame cannot tell you that a person is running (versus standing with one leg raised), that a car is decelerating (versus at rest), or that a conversation is becoming heated (versus a momentary pause). Understanding video requires modeling change over time, and the computational cost of doing so at scale is substantial.

The temporal dimension does not simply add one more feature to process. It introduces a qualitatively new type of information: causality. Two frames that look similar can be part of very different events depending on what happened in the frames between them. A person with arms raised might be celebrating a goal, surrendering, or reaching for something on a high shelf. Disambiguating these requires understanding the sequence of events leading up to the current frame alongside the current frame itself. This dependency on temporal context fundamentally shapes how video understanding systems must be designed and evaluated.

Early video understanding models extended 2D convolutional networks to 3D by adding a temporal convolution dimension, creating 3D ConvNets (C3D, I3D) that could detect spatiotemporal features like motion edges and action-specific appearance changes. These architectures worked for action recognition tasks on short clips but were computationally expensive and required large labeled video datasets to train from scratch. Their parameter efficiency was poor: a 3D convolution kernel has one more dimension than its 2D counterpart, multiplying parameter count proportionally with kernel temporal size.

The transformer era brought video transformers (ViViT, TimeSformer) that tokenize video frames into patches and apply attention across both spatial and temporal dimensions. This is conceptually straightforward but computationally challenging. Full spatiotemporal attention computes attention between every patch in every frame and every other patch in every frame, which has complexity $O((N \cdot T)^2)$ where $N$ is the number of patches per frame and $T$ is the number of frames. For a 10-second video at 30 frames per second with 14x14 patches per frame, this produces roughly $176$ billion attention pairs, which is completely intractable on current hardware.

Factored attention addresses this by separating spatial and temporal attention into alternating layers. A spatial attention layer computes attention only among patches within the same frame, independently for each frame. A temporal attention layer computes attention only among the same spatial position across different frames, independently for each spatial location. This reduces complexity to $O(N^2 T + N T^2)$, which is manageable for videos of moderate length.

The factored attention formula for a patch at spatial position $(i, j)$ in frame $t$ is computed in two stages:

$$\mathbf{z}^{\text{spatial}}_{i,j,t} = \text{Attention}\left(\mathbf{q}_{i,j,t}, \{\mathbf{k}_{i',j',t}\}_{i',j'}, \{\mathbf{v}_{i',j',t}\}_{i',j'}\right)$$ $$\mathbf{z}^{\text{temporal}}_{i,j,t} = \text{Attention}\left(\mathbf{q}_{i,j,t}, \{\mathbf{k}_{i,j,t'}\}_{t'}, \{\mathbf{v}_{i,j,t'}\}_{t'}\right)$$

where:

• $\mathbf{q}_{i,j,t}$: the query vector for the patch at spatial position $(i,j)$ in frame $t$
• The spatial attention equation: keys and values come from all positions $(i', j')$ within the same frame $t$, letting the model to integrate context within a single moment
• The temporal attention equation: keys and values come from the same spatial position $(i,j)$ across all frames $t'$, letting the model to track how a specific location changes over time

The factored approach makes an implicit assumption that has both a cost and a benefit. The cost is that spatial context and temporal context cannot be jointly considered in a single attention operation: a patch cannot attend simultaneously to a different spatial location and a different time. The benefit is that this factored structure matches certain natural properties of video. Spatial relationships (object layout, scene structure) change relatively slowly, while temporal relationships (motion, causality) can be tracked by following specific locations through time. By separating these two concerns, factored attention allocates its computational budget efficiently.

Before a video model can process any frames, it must decide which frames to process. The naive answer is to sample uniformly, but uniform sampling has a significant weakness: most of the information in many videos is concentrated in short temporal windows surrounding key events, with long stretches of relatively static content in between. A 30-second cooking video might have 20 seconds of chopping (repetitive, uniform content) and 10 seconds of necessary technique (varied, informative content). Uniform sampling allocates frames equally across both, under-representing the necessary moments.

Content-adaptive sampling strategies try to allocate frame budget to moments of high information content. The simplest version computes the L2 distance between consecutive frame embeddings and samples more densely near local maxima of this distance signal. A scene cut produces a large spike in frame-to-frame distance; a period of slow camera pan produces low distance values. Sampling adaptively near these spikes captures scene transitions, action onsets, and key events more reliably than uniform sampling.

More advanced approaches use learned keyframe extractors that are trained on downstream task supervision. For a video question answering system, the keyframe extractor learns to select frames that are most informative for answering the types of questions in the training set. This is a form of differentiable frame selection that jointly optimizes keyframe identification and question answering, though training such systems end-to-end requires careful design to keep gradients flowing through the discrete selection operation.

Video question answering (VideoQA) extends image-based VQA to the temporal domain. A model watches a video clip and answers questions that may require understanding temporal sequences, counting events, or reasoning about causality. "What happened immediately before the person fell?" requires identifying the temporal event boundaries and reasoning about the causal chain. "How many times did the player touch the ball?" requires counting events across the full video duration, not just identifying that an event happened at all.

Benchmarks like NExT-QA and STAR specifically target causal and temporal reasoning in video, in contrast to earlier benchmarks like MSVD-QA and MSRVTT-QA that could often be answered by looking at a single representative frame. The causal and temporal questions in NExT-QA require models to understand that some questions can only be answered by examining how the video changes over time, not just what it looks like at any moment.

Modern VideoQA models typically sample a fixed number of frames from the video (often 8 to 32, depending on video length and computational budget), encode each frame independently using a vision encoder, and then use cross-frame attention to aggregate temporal context before generating the answer. The cross-frame attention step is where temporal reasoning happens: the model learns to attend to the frames most relevant to the question and to reason about their temporal ordering.

A revealing diagnostic for VideoQA systems is comparing performance on questions answerable from a single frame versus questions requiring temporal integration. Many early VideoQA models showed negligible improvement from temporal modeling, suggesting they were largely solving the task from static frame content. Models specifically designed for temporal reasoning, with attention mechanisms that operate across frames and training objectives that require temporal understanding, show larger improvements on the temporally-demanding subsets of these benchmarks.

Most video understanding benchmarks focus on short clips of 10 to 30 seconds. Real-world applications often require understanding videos that are minutes or hours long: analyzing a surgical procedure, summarizing a lecture, understanding a documentary, reviewing surveillance footage. At these scales, even sampling 32 frames per video misses the majority of content, and the important moments may be scattered throughout the full duration with no obvious prior way to identify their location.

Long-form video understanding is an active research area with several competing approaches. Memory-augmented models maintain a compact summary of previously seen frames that is updated as new frames arrive. Each new frame is processed in the context of this running summary, which allows the model to integrate information from across the full video duration without storing all frames in memory. The challenge is that the memory must be compact enough to fit in the context window but rich enough to preserve information that will be relevant for future reasoning.

Hierarchical models first perform coarse-grained segment-level analysis to identify relevant portions of the video, then apply fine-grained analysis only to those portions. For a question about a specific event in a long video, the coarse-grained pass might identify the 2-minute segment most likely to contain the relevant content, and the fine-grained pass then processes only those 2 minutes at full resolution. This hierarchical strategy can reach near-full-resolution understanding of specific moments with a small fraction of the computational cost of processing the entire video.

Retrieval-based approaches embed all frames in a shared embedding space and use approximate nearest-neighbor search to retrieve the frames most semantically similar to a given query before running the full model. For a query about a specific event, retrieval narrows the processing to a small subset of relevant frames, sharply reducing the effective computational cost. The Goldfish model and similar work on Needle-in-a-Haystack video benchmarks show that retrieval-based approaches reach strong performance on queries requiring specific information in hours-long videos, while reducing computation by orders of magnitude compared to processing all frames uniformly.