Geometry, Mapping, and 3D/4D State › What Geometry Contributes to a World Model

Before closing, it is worth assembling the ledger. Geometry is not one technique but a set of distinct contributions to a world-modeling system, and being precise about which one you are using clarifies both design and evaluation. The same geometric machinery shows up in four different roles, and confusing the roles leads to characteristic mistakes: evaluating a sensor model as if it were a transition model, or expecting a state format to carry its own dynamics.

• Geometry as state: an allocentric map can describe scene geometry in a world frame that is invariant to egomotion, while camera-frame depth and egocentric occupancy change equivariantly under the known frame transform. Both forms can contribute to the control-sufficient state discussed in Part III, provided the coordinate frame is explicit. Moving the robot translates and rotates an egocentric map predictably rather than producing arbitrary view-dependent changes. This equivariance should not be confused with controllability in control theory, which asks whether admissible inputs can steer a system between states. Separating the persistent world-frame component from the known observation or frame transform lets a learned transition describe scene evolution without absorbing egomotion into the dynamics. The full agent-world state still includes the agent's evolving pose and action effects.
• Geometry as observation model: geometric state paired with sensor geometry and a neural-field, mesh, or splat renderer can make a parameterized observation prediction differentiable and serve as the forward model inside $p(\mathbf{o} \mid \mathbf{s})$. Given a hypothesized state, you can render what the sensor should see and compare it with the observation. With a specified likelihood, prior, and parameterization, gradients through that discrepancy support approximate maximum-likelihood or MAP inference over state. Ambiguities, local minima, model mismatch, and the choice of loss still determine what the optimization recovers.
• Geometry as a training signal: multi-view consistency, reprojection error, and photometric reconstruction provide self-supervised objectives without semantic labels. Failure to render a consistent shifted view shows that the combined geometry-and-observation model is inadequate, although the cause may lie in geometry, appearance, visibility, calibration, renderer capacity, or optimization. Next-frame prediction and cross-view consistency can therefore come apart: a video predictor may capture appearance and motion well enough for one-step forecasts while remaining inconsistent across viewpoints. Under adequate parallax and valid image-formation assumptions, reprojection is a complementary test that is often especially diagnostic of cross-view structure. It is not a proof of unique or correct geometry.
• Geometry as a planning substrate: collision checking, reachability, visibility reasoning, and grasping all operate on geometric state. A prediction of "the chair will be at pose P in 3 seconds" gives a motion planner a direct spatial representation. The planner can query trajectory intersections, grasp reachability, and visibility through the representation and its spatial index, often without routing each decision through image synthesis and perception. Exact cost still depends on the representation, collision model, and planner architecture.

The recurring theme is the distinction we have drawn since Part I: open-loop prediction quality and closed-loop decision usefulness come apart. A photorealistic video prediction can score well while being geometrically inconsistent across views, making it unsuitable for metric path planning even though its usefulness for other decisions remains an empirical question. Conversely, a coarse occupancy forecast that is geometrically consistent may enable reliable navigation despite looking crude. Predictive fidelity, representation quality, uncertainty quality, and decision usefulness are different axes. Explicit geometry offers unusually direct control over them: sharper reconstruction for representation quality, occupancy probabilities and covariances that can be evaluated for calibration, differentiable rendering for predictive fidelity, and plan-compatible state for decision usefulness.