Geometry, Mapping, and 3D/4D State › Limitations and Impact

The geometric toolkit comes with assumptions that fail in characteristic ways, and knowing the failure modes is part of the competence.

One of the most consequential assumptions in the methods discussed here is a static world. Occupancy grids accumulate evidence indefinitely, which is only sound if cells do not change. In dynamic environments, a naive static-world SLAM pipeline may discard moving points as outliers or, worse, bake them into the map as ghost geometry. Dynamic and hybrid systems instead detect, mask, track, or explicitly model moving objects. Ghost geometry is nastier than missing geometry when a planner trusts it and routes around objects that are no longer there. Dynamic-scene SLAM and 4D reconstruction address this, but they reintroduce the segmentation problem: separating egomotion, object motion, and deformation is impossible without committing to some structure, and the wrong commitment produces confident nonsense. If you assume two rigid bodies where the world has one deformable one, limited observations may let the estimator report two objects moving in a coordinated way. The explanations can be observationally ambiguous, although boundary discontinuities, correlated residual structure, additional views, and explicit model comparison may expose the mismatch. Relatedly, the independence assumptions that make grid filters tractable, cells independent and measurements independent given the map, are violated by correlated sensors and correlated errors. Real range sensors can have systematic calibration, timing, pose, or environmental biases that correlate many or all rays in a scan, and neighboring cells are physically part of the same surfaces. Drift is accumulated estimation error, especially pose error produced by sequential integration. Violated independence assumptions and correlated sensor biases can contribute to drift, but are not synonymous with it. A valid loop closure can constrain accumulated trajectory drift; it does not repair every local sensor, calibration, semantic, or model error.

Second, geometric fidelity is expensive and its value is task-dependent. Dense metric reconstruction of an entire environment at centimeter resolution is far more memory, compute, and calibration burden than many tasks need, while some tasks (manipulation under occlusion, liquid handling, deformable objects) need geometry richer than any of these representations provide. There is a real engineering judgment call in choosing the coarsest geometric state that is sufficient for the decisions at hand, echoing the state-abstraction arguments of Part III. Overshooting buys cost; undershooting buys plans that fail on contact. The judgment is made harder by the fact that sufficiency is not static: a navigation task may need only coarse occupancy today, but the same robot asked to open a drawer tomorrow needs contact-accurate geometry of the drawer's handle, so the choice of geometric state is a bet about the task distribution the agent will face.

Third, the semantic and geometric layers can disagree. Despite a substantial literature on joint fusion, conflicts remain unresolved in important regimes. A semantic map that classifies a cell as "floor" while geometry says it is a 30-centimeter step has a conflict that matters to a wheeled robot, and its resolution remains open: the classifier may be wrong, the geometry may be wrong, or the category "floor" may simply be too coarse for the decision at hand. Learned monocular components remain subject to geometric scale non-identifiability unless metric supervision or another prior supplies a scale anchor. A feed-forward 3D reconstruction without such an anchor can be unusable for metric navigation even when it looks plausible; learned metric priors can select a scale, but the pixels alone do not identify it. Plausible-looking reconstructions can hide severe metric errors, so visual inspection is not sufficient validation.

None of this diminishes the impact. Geometry provides a longstanding, widely used bridge between raw high-dimensional observations and compact, action-relevant state. It also supplies self-supervised objectives such as multi-view consistency and reprojection, and makes image-formation models explicit and differentiable so perception and prediction can be trained together. Modern learned systems reuse analogous structures: egomotion-aware latents parallel geometric state, cross-view losses use projection and correspondence constraints, and learned renderers build on differentiable image-formation models. Evaluation follows directly from the four roles: reprojection and novel-view consistency for the observation model, absolute trajectory error and relative pose error for localization, reconstruction IoU or depth accuracy for mapping, and, decisively, downstream task performance for whether the geometric state was sufficient. A model can be excellent on the first three and still fail the fourth if the geometry it chose to represent is not the geometry the task depends on.