Geometry, Mapping, and 3D/4D State › Summary

• Projection destroys depth: a pinhole camera maps each visible 3D point to a pixel; conversely, each pixel back-projects to a ray, so a single image leaves an independent depth ambiguity along each ray. Calibrated rigid monocular multi-view reconstruction instead has a shared global-scale ambiguity. Multiple views, active sensing, or learned priors can supply depth information; a learned prior may predict metric scale without making it geometrically identifiable from the image alone.
• Rigid transforms and homogeneous coordinates are the bookkeeping that connects world, body, and sensor frames. A rigid camera-to-body mount transform is fixed calibration; composing it with the evolving body pose gives the world-to-camera transform, which changes as the agent moves. Intrinsics are often treated as fixed for a stable imaging configuration but must be updated when image formation changes. The homogeneous form turns transform composition and inversion into ordinary matrix operations, making frame trees manageable in practice.
• Depth, pose, and scene flow are coupled estimation problems. Decomposing scene flow into a rigid (egomotion or object) component plus a residual separates camera-induced change from real dynamics and can cue object segmentation. With correct, well-distributed matches and sufficiently independent unbiased errors, the rigid fit can also average noise across points.
• SLAM treats map and trajectory as one joint inference problem. EKF-SLAM makes the dense coupling explicit; sparse graph formulations and incremental solvers scale estimation, while loop closures add long-range constraints that can correct accumulated drift. Occupancy grids accumulate per-cell Bernoulli beliefs in log-odds form and explicitly represent unknown space, at the price of static-world and independence assumptions. Preserving unknown space enables uncertainty-aware objectives to drive exploration.
• Neural fields expose continuous coordinate functions, often replacing dense uniform output grids while sometimes using sparse or multiresolution grids internally. Fourier features can change the optimization bandwidth used to address spectral bias; NeRF-family methods and Gaussian splatting make the sensor model differentiable and encourage multi-view consistency; 4D extensions add time through deformation fields or time-conditioned parameters, with canonical-plus-warp providing a particularly structured neural-field treatment relative to direct time conditioning.
• Keep the roles distinct: a geometric reconstruction is a state format. Paired with an explicit sensor model or renderer, it supports an observation model. It is not, by itself, a transition model. A system that renders the past beautifully has not yet earned the right to be called a world model.

Geometry tells us what is where and can describe observed motion through scene flow or time-indexed 4D structure. Geometry alone does not specify the action-conditioned dynamics needed to predict counterfactual futures: whether the stacked boxes will topple after a push, how the ball will bounce under a changed action, or where disturbed water will flow. That is the domain of physical dynamics and constraints, and it is exactly where we go in the next chapter.