Geometry, Mapping, and 3D/4D State › Geometry, Mapping, and 3D/4D State

Imagine a delivery robot entering a cluttered lobby. Two cameras on its head see a chair from slightly different viewpoints, producing two very different arrays of pixels. A second later, the robot turns, and the pixel array changes extensively again. Yet the chair itself has not moved. If the robot's world model stores predictions over pixels, almost every "state" it tracks changes whenever the agent so much as rotates its head. If instead it tracks where the chair is, in a frame that does not move with the robot, the chair's physical pose stays stable while observations flow past it. That pose changes when something in the world acts on the chair; the robot's belief about the pose can also change when new evidence arrives. Separating physical state from belief state is not an aesthetic preference. It lets physical transitions represent events in the world, belief updates represent acquired information, and neither be confused with artifacts of the observer's own motion.

This is the core promise of geometric state. Building on the discussion of sufficient state from Part III, geometry provides allocentric representations that can remain fixed under egomotion and egocentric representations that transform equivariantly under known pose changes. Building on the body schema from Part III, the agent's own pose becomes part of the state, the hinge that connects "where I am" to "what I see." Pose lets the agent factor the geometric-motion component of observation change into two causes: the observer moved, or the scene moved. Illumination, visibility, exposure, calibration, sensor noise, and other image-formation effects remain additional causes of changing observations. Separating observer motion from scene motion is a longstanding organizing idea in robotics perception, and it remains useful when learned world models make frames and action effects explicit. Geometry serves four roles here: a state format, a sensor model, a source of self-supervised training signal, and a practical substrate for planning. These roles connect influential classical estimators with influential learned geometric representations.

A note on what "better" means here, because it will recur throughout. Geometric representations can be evaluated along four partly independent axes. Predictive fidelity: does the model reproduce future observations accurately? Representation quality: does the internal 3D structure match the true scene structure, even where no camera ever looked? Uncertainty quality: do the confidences attached to geometric estimates (depth variances, occupancy probabilities, pose covariances) accurately reflect what is and is not known? Decision usefulness: can a planner consume the predictions and act well? A representation can excel on one axis and fail another. Photorealistic view synthesis can have high predictive fidelity with poor uncertainty quality; a blurry but well-calibrated occupancy map may be far more decision-useful than a sharp overconfident one. Keeping these axes separate is the discipline this chapter tries to enforce.

In this chapter we work through the geometric toolkit in four layers. First, coordinate frames and camera geometry: how a 3D world projects into 2D observations, and what that projection destroys. Second, depth, pose, and scene flow: how to recover 3D quantities from observations. Third, SLAM, occupancy grids, and semantic maps: how to accumulate geometric state over time under uncertainty. Fourth, neural fields and dynamic 3D representations: how modern continuous, learnable scene representations replace and extend classical grids. Throughout, we distinguish what a geometric model predicts from how a decision-making agent uses those predictions.