Geometry, Mapping, and 3D/4D State

Imagine a delivery robot entering a cluttered lobby. Two cameras on its head see a chair from slightly different viewpoints, producing two very different arrays of pixels. A second later, the robot turns, and the pixel array changes extensively again. Yet the chair itself has not moved. If the robot's world model stores predictions over pixels, almost every "state" it tracks changes whenever the agent so much as rotates its head. If instead it tracks where the chair is, in a frame that does not move with the robot, the chair's physical pose stays stable while observations flow past it. That pose changes when something in the world acts on the chair; the robot's belief about the pose can also change when new evidence arrives. Separating physical state from belief state is not an aesthetic preference. It lets physical transitions represent events in the world, belief updates represent acquired information, and neither be confused with artifacts of the observer's own motion.

This is the core promise of geometric state. Building on the discussion of sufficient state from Part III, geometry provides allocentric representations that can remain fixed under egomotion and egocentric representations that transform equivariantly under known pose changes. Building on the body schema from Part III, the agent's own pose becomes part of the state, the hinge that connects "where I am" to "what I see." Pose lets the agent factor the geometric-motion component of observation change into two causes: the observer moved, or the scene moved. Illumination, visibility, exposure, calibration, sensor noise, and other image-formation effects remain additional causes of changing observations. Separating observer motion from scene motion is a longstanding organizing idea in robotics perception, and it remains useful when learned world models make frames and action effects explicit. Geometry serves four roles here: a state format, a sensor model, a source of self-supervised training signal, and a practical substrate for planning. These roles connect influential classical estimators with influential learned geometric representations.

A note on what "better" means here, because it will recur throughout. Geometric representations can be evaluated along four partly independent axes. Predictive fidelity: does the model reproduce future observations accurately? Representation quality: does the internal 3D structure match the true scene structure, even where no camera ever looked? Uncertainty quality: do the confidences attached to geometric estimates (depth variances, occupancy probabilities, pose covariances) accurately reflect what is and is not known? Decision usefulness: can a planner consume the predictions and act well? A representation can excel on one axis and fail another. Photorealistic view synthesis can have high predictive fidelity with poor uncertainty quality; a blurry but well-calibrated occupancy map may be far more decision-useful than a sharp overconfident one. Keeping these axes separate is the discipline this chapter tries to enforce.

In this chapter we work through the geometric toolkit in four layers. First, coordinate frames and camera geometry: how a 3D world projects into 2D observations, and what that projection destroys. Second, depth, pose, and scene flow: how to recover 3D quantities from observations. Third, SLAM, occupancy grids, and semantic maps: how to accumulate geometric state over time under uncertainty. Fourth, neural fields and dynamic 3D representations: how modern continuous, learnable scene representations replace and extend classical grids. Throughout, we distinguish what a geometric model predicts from how a decision-making agent uses those predictions.

This section covers the vocabulary that everything else depends on: frames of reference, rigid transforms, and the pinhole camera model that connects 3D points to pixel observations. The material is classical, but it deserves careful treatment, because many practical failures in 3D perception and learned world modeling trace back to frame bookkeeping done carelessly: a rotation applied in the wrong direction, a translation expressed in the wrong frame, a depth confused with a range.

A frame is a coordinate system attached to something: the world (allocentric), the robot's body (egocentric), a camera, an object. Every geometric statement is implicitly relative to a frame, and much of practical 3D robotics is bookkeeping about which frame each quantity lives in. The phrase "the chair is at position $(1.2, 0.5, 0)$" is incomplete on its own; it only becomes a claim about the world once we say whose coordinates those are. In a world model, this incompleteness is dangerous, because learned networks can silently internalize whatever frame dominated their training data, and then fail unpredictably when the deployment frame convention differs.

Rigid transforms move points between frames. A rigid transform is a rotation followed by a translation; it preserves distances and angles. Physically, this is exactly the transformation relating any two frames that are bolted to rigid bodies, which is why it suffices for cameras mounted on robots, hands holding objects, and sensors on vehicles. We write the transform from world frame $W$ to camera frame $C$ as:

$$\mathbf{x}_C = R_{CW} \mathbf{x}_W + \mathbf{t}_{CW}$$

where $R_{CW} \in SO(3)$ is a $3 \times 3$ rotation matrix ($R^\top R = I$, $\det R = 1$) and $\mathbf{t}_{CW}$ is a translation vector. The two constraints defining $SO(3)$ have physical meaning: orthogonality says the camera's axes stay perpendicular, and unit determinant says the axes keep a right-handed orientation, which excludes reflections. The subscript convention $R_{CW}$ reads naturally as "the rotation taking points expressed in $W$ into points expressed in $C$." The set of all such transforms forms the group $SE(3)$, and composing them is just matrix-style chaining: walking from the camera frame to the body frame to the world frame composes two transforms into one. This composition property is what makes transform graphs workable in practice. A robot typically maintains a tree of frames, camera-to-gripper, gripper-to-wrist, wrist-to-base, base-to-world, and any point can be moved between any two frames by composing the transforms along the connecting path.

The awkward part of this notation is that rotations and translations do not combine into a single matrix multiplication on 3-vectors. Composition becomes messy: $(R_2, \mathbf{t}_2) \circ (R_1, \mathbf{t}_1) = (R_2 R_1, R_2 \mathbf{t}_1 + \mathbf{t}_2)$, and inverting means remembering that the translation must be rotated back before being negated. Homogeneous coordinates fix all of this. We append a 1 to every point, making it a 4-vector, and write the transform as one $4 \times 4$ matrix:

$$\tilde{\mathbf{x}}_C = T_{CW} \tilde{\mathbf{x}}_W, \qquad T_{CW} = \begin{bmatrix} R_{CW} & \mathbf{t}_{CW} \\ \mathbf{0}^\top & 1 \end{bmatrix}$$

Now composition is ordinary matrix multiplication, $T_{AC} = T_{AB} T_{BC}$, and inversion is (mostly) a structured transpose-like operation: for a rigid transform, $T^{-1} = \begin{bmatrix} R^\top & -R^\top \mathbf{t} \\ \mathbf{0}^\top & 1 \end{bmatrix}$, which needs only the already-available $R^\top$ rather than a general matrix inverse. When the transform is a pure frame-to-frame map like $T_{CW}$, the inverse is the reverse map $T_{WC}$; software can obtain it with the structured rigid-transform inverse or cache it as an implementation choice. Homogeneous coordinates also let us treat points at infinity uniformly, which matters because projection is about to send finite 3D points onto a 2D plane where direction, not distance, survives. A direction is represented by a homogeneous vector whose final coordinate is zero. Under projection, a family of parallel 3D lines meets at a vanishing point, while the directions contained in a plane form a horizon line. The projective framework handles these ideal elements with the same algebra as ordinary points rather than as special cases.

The pinhole model is the workhorse idealization of a camera. Light passes through a single point (the optical center). The physical sensor lies behind that center and forms an inverted image. The conventional computer-vision model used below instead intersects rays with a virtual image plane in front of the center; the image-coordinate convention absorbs the physical inversion and gives the positive-sign projection equations. Lens distortion is a common geometric departure that calibration can estimate, but real image formation also includes finite-aperture blur, aberrations, motion blur, rolling shutter, sampling, exposure, and other effects that may need explicit models for the task at hand. An ideal visible 3D point in camera coordinates $\mathbf{x}_C = (X, Y, Z)^\top$, with positive optical-axis depth $Z > 0$, projects to pixel coordinates:

$$u = f_x \frac{X}{Z} + c_x, \qquad v = f_y \frac{Y}{Z} + c_y$$

The equations have a direct physical reading. Dividing by $Z$ is the perspective effect: for fixed lateral coordinates $X$ and $Y$, increasing positive depth moves the projection toward the principal point. The focal lengths $f_x, f_y$ scale normalized ray coordinates $X/Z$ and $Y/Z$, the tangents of the off-axis ray angles, into pixels. Treating pixel offset as directly proportional to angle is a small-angle approximation. A longer focal length means a given lateral offset $X$ at depth $Z$ produces a larger pixel displacement. The principal point $(c_x, c_y)$ accounts for the fact that the optical axis does not usually pierce the sensor exactly at its center. In matrix form, stacking the whole pipeline from world to pixel:

$$\tilde{\mathbf{u}} \sim K \begin{bmatrix} R & \mathbf{t} \end{bmatrix} \tilde{\mathbf{x}}_W, \qquad K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}$$

This compact expression is worth internalizing because it separates cleanly: the $3 \times 4$ matrix $[R \mid \mathbf{t}]$ does the world-to-camera rigid motion, the calibration matrix $K$ does the camera-to-pixel mapping, and the result is a single linear operation in homogeneous coordinates. Homogeneous coordinates isolate this linear projective transform from the nonlinear perspective division that follows; real cameras add lens distortion and other nonidealities outside the displayed model.

The tilde and the $\sim$ carry real meaning: $\tilde{\mathbf{u}}$ is a homogeneous 3-vector, and $\sim$ means equality up to multiplication by any nonzero scale. Converting that homogeneous image vector to ordinary pixel coordinates then requires dividing by its third coordinate. That division is the nonlinear heart of perspective, and it is where information dies. Up to that division, the mapping from world point to image point is linear, which is why much of multi-view geometry can be developed with linear algebra, and why the division step is exactly where reconstruction difficulty concentrates.

Look at what the projection discards. The pixel $(u, v)$ depends only on the direction of $\mathbf{x}_C$, not its length. The camera-frame points $(X,Y,Z)=(1,0,2)$ and $(2,0,4)$ land on exactly the same pixel because both $X/Z$ and $Y/Z$ match. The perspective division by $Z$ has erased the overall scale of the camera-frame point. Every observation is a ray from the optical center, and the observation alone cannot tell you where along the ray the world is. Formally, the two pixel coordinates provide two scalar constraints on a three-dimensional point, leaving a one-parameter family of locations along the ray. This single fact drives most of the rest of the chapter: recovering depth requires either extra information (a second viewpoint, a depth sensor, a learned prior) or extra assumptions. Whenever a system appears to get depth from a single image for free, one of these three is hiding somewhere, and it is worth finding out which.

To see the collapse concretely, the next cell places several points along a single viewing ray at different depths, projects each one, and also compares axial depth against Euclidean range for off-axis points.

Also note that $Z$ is depth along the optical axis, not Euclidean distance to the point. Conflating the two is a common source of subtle bugs. For a point well off-axis, the Euclidean range can be substantially larger than the axial depth, and a stereo algorithm, a lidar, and a learned depth network may each report a slightly different one of these quantities by convention. When fusing sensors or comparing predictions against ground truth, always check which convention each source uses.

Given the ray ambiguity above, how do agents recover depth? There are three broad strategies, and they differ in what they assume and what they guarantee. It is worth being explicit about the source of depth information in each case, because that source determines the failure modes.

• Stereo and multi-view geometry: two cameras with a known baseline see the same point from two positions. The point's depth is the only value consistent with both observations. Geometrically, each pixel defines a ray. For a correct correspondence from compatible calibrated pinhole cameras, noise-free rays meet at the scene point in a nondegenerate configuration. For a rectified pair with parallel optical axes and a horizontal baseline $B$, horizontal disparity $d$ and focal length $f$ in pixels obey $Z = f B / d$. Arbitrary calibrated viewpoints instead require triangulation from their projection matrices. The rectified-pair inverse relationship has important operational consequences. A larger baseline improves first-order depth precision when focal length in pixels and disparity uncertainty are controlled. More pixels across the same field of view can also help when optical quality, calibration, correspondence quality, and disparity uncertainty in pixel units do not degrade. Resolution alone cannot recover information lost to blur or matching error. Distant points have tiny disparity and are hard to localize, which is why stereo depth maps are accurate nearby and noisy in the distance.

The inverse relationship is easiest to trust when plotted: the next cells evaluate the stereo equation over a depth range for two baselines.

Recovering geometry from observations involves three coupled estimation problems: depth (where points are), pose (where the sensor or object is), and scene flow (how points move). They are coupled because you usually need one to estimate the others: motion of the camera explains part of the image change, motion of the world explains the rest. To convert a set of depth measurements into a world-frame map you need the sensor pose; to estimate the sensor pose from point clouds you need the geometry of the surfaces around you; to tell whether a surface moved you need to know how much of its apparent motion came from the sensor. There is no fixed order of operations that breaks this loop cleanly. Practical systems break it with approximations, iteration, or priors, and the quality of the resulting geometry depends on how honest the system is about which variable was held fixed while estimating the others.

This section covers two common pose-estimation cases, distinguished by the correspondences available: 3D-to-2D and 3D-to-3D.

• 3D-to-2D (perspective-n-point): given known 3D points and their pixel projections, solve for the camera pose. Each known 3D point together with its observed pixel constrains the pose, because the point must lie on that pixel's ray, and enough rays must intersect at a common camera position and orientation. This is what happens when a robot localizes against a known map, or when it estimates an object's pose from detected keypoints. For a calibrated perspective camera, P3P uses three nondegenerate correspondences but can yield multiple pose solutions, so a fourth correspondence is commonly used to disambiguate them. Other camera models and degenerate point configurations change the requirement. Closed-form and iterative solvers exist, and robust variants handle outliers. In deployed pose estimation, robustness to correspondence outliers is often at least as important as nominal inlier accuracy: a single mis-detected keypoint, if trusted, can drag the whole pose estimate far away, so deployed systems pair the solver with RANSAC-style hypothesis testing that fits the pose to random minimal subsets and keeps the one supported by the most correspondences.
• 3D-to-3D (Procrustes / Kabsch): given two point clouds that are related by a rigid transform, the best-fit rotation and translation have a closed-form solution via SVD. This is the core step inside iterative closest point (ICP) registration and inside many odometry pipelines. ICP illustrates how a chicken-and-egg problem is handled by alternation: correspondences between the clouds are guessed from the current transform estimate, the optimal transform for those correspondences is computed in closed form, and the two steps repeat until convergence. Each half of the loop is easy precisely because the other half is temporarily fixed, an alternation also used by expectation-maximization. An EKF addresses coupling differently: it performs sequential prediction and a joint linearized Bayesian update rather than alternating between fixed optimization blocks.

The Kabsch solution is worth knowing because it recurs everywhere. Given matched source points $P$ and target points $Q$, first compute their centroids and the centered versions $\tilde{P} = P - \bar{P}$ and $\tilde{Q} = Q - \bar{Q}$:

1. Compute the cross-covariance $H = \tilde{P}^\top \tilde{Q}$.
2. Take the SVD $H = U \Sigma V^\top$.
3. The optimal rotation is $R = V D U^\top$, where $D$ corrects the sign if $\det(VU^\top) < 0$ (reflections are not rigid transforms).
4. The translation is $\mathbf{t} = \bar{Q} - R \bar{P}$.

A few remarks on why this works and when it does not. Centering both clouds first removes the translation from the problem, leaving a pure rotation to be found; the cross-covariance $H$ summarizes how source directions map to target directions, and the SVD extracts the orthogonal matrix best aligning them. The determinant correction in step 3 is essential: without it, the least-squares optimum occasionally wants to be a reflection, which is geometrically illegal, and the correction projects back to the nearest legal rotation. Finally, the closed form is only as good as its correspondences. Kabsch assumes each source point is already matched to the correct target point; if the matching is wrong, the transform will be confidently wrong, which is why ICP wraps this solver in an outer loop that revises the matching.

Scene flow is where geometry meets dynamics, and it deserves careful decomposition. When a sensor moves through a static world, all apparent sensor-coordinate change is explained by one rigid transform: the egomotion. This is a remarkably strong structural prior, because a single $SE(3)$ transform accounts for the coordinate change of potentially millions of observed points at once. Camera and object rigid transforms compose rather than adding as independently defined displacements. A precise destination-frame residual transports the time-$t$ point from $C_t$ into $C_{t+1}$ before subtraction:

$$\mathbf{f}^{\mathrm{obj},C_{t+1}}_i := \mathbf{p}^{C_{t+1}}_i(t{+}1) - \left(T_{C_{t+1}C_t}\,\tilde{\mathbf{p}}^{C_t}_i(t)\right)_{1:3},$$

where the homogeneous transform is applied before taking the three-vector difference. A static world point has zero residual under the correct egomotion transform; nonzero residual captures object motion or modeling error in $C_{t+1}$.

A world model without an explicit separation must learn observer-induced change jointly with scene dynamics. It may misattribute that change or generalize poorly unless action or pose conditioning, model capacity, data, and observability make egomotion predictable. Explicitly decomposing a global rigid component from the residual is therefore a useful sanity check in dynamic-scene perception, not a prerequisite for every successful predictor. The residual contains the dynamics of walking people, articulating arms, and falling objects, along with any uncompensated estimation error. It is also, as you might expect from Part III's object-centric chapter, a strong cue for segmentation: points that share a residual motion tend to belong to the same object. Coherent residual motion can support segmentation before object classification, although it does not by itself establish object identity.

Let's estimate a rigid transform from a moving point cloud and look at what remains.

A word of caution: this decomposition assumed the sensor was static. In practice the sensor moves, and egomotion must be estimated and removed first, which is exactly the localization problem of the next section. Errors in egomotion leak directly into false "object motion." If the estimated camera motion is wrong by a small rotation, every static point acquires a small coherent residual that can masquerade as a slow-moving object. This coupling is why dynamic-scene perception and localization are usually solved together or in an alternating scheme, and why any claim of "object detection by motion" should be checked against the quality of the egomotion estimate underneath it.

This section covers how an agent builds and maintains geometric state over time: the coupling between localization and mapping, the classical EKF solution, occupancy grids for dense maps, and semantic extensions. The unifying theme is accumulation under uncertainty: single-frame estimates are noisy and partial, and the SLAM family of methods is about fusing many of them into a consistent whole without letting errors compound.

To build a map from a moving sensor, you need to know where each observation was taken. To know where you are, it helps to have a map to match against. This chicken-and-egg structure is the defining feature of SLAM: simultaneous localization and mapping. It is worth spelling out why neither half alone suffices. Odometry, the integration of motion measurements, gives localization without a map, but without an absolute or map-relative correction its uncertainty accumulates and dead-reckoning drift is generally unbounded over long horizons. Individual zero-mean errors can cancel on a particular realization; it is the growing uncertainty, and systematic bias when present, that makes uncorrected odometry unreliable. Matching against a map gives map-relative localization that can bound or correct accumulated odometric drift, though map, calibration, association, and environmental errors remain. In an unknown environment without an external absolute reference such as a surveyed map, GNSS, motion capture, or known landmarks, the SLAM solution is to estimate pose and map together and let each constrain the other. Formally, the agent wants the joint posterior:

$$p(\mathbf{x}_{1:t}, m \mid \mathbf{z}_{1:t}, \mathbf{u}_{1:t})$$

over the entire trajectory $\mathbf{x}_{1:t}$ and the map $m$, given all observations and controls. Full-trajectory SLAM is a smoothing problem because today's map depends on where the robot was yesterday. EKF-SLAM, by contrast, is an online filtering formulation over the current pose and persistent map state. When the robot realizes, upon recognizing a landmark, that its pose from an hour ago was wrong, every observation taken since then was attached to the map at the wrong place and must be corrected retroactively. This is why SLAM is fundamentally harder than the filtering problems of Part II: the state whose history matters keeps growing. In Part II we met Bayesian filtering over a belief state; SLAM is what happens when the belief state grows to include a model of the world itself, and the state and the map become statistically coupled through every shared observation. The connection to world models is direct: SLAM is the classical instance of maintaining a persistent, action-conditioned belief over scene structure, updated by every observation and corrected by every act of re-observation.

The extended Kalman filter approach makes the belief explicit. For the planar example below, the robot pose is $\mathbf{x}=(x,y,\theta)$ and each landmark $\mathbf{l}_j$ is a point in $\mathbb{R}^2$. The state vector stacks that pose and all landmark positions:

$$\mu = \begin{bmatrix} \mathbf{x} \\ \mathbf{l}_1 \\ \vdots \\ \mathbf{l}_M \end{bmatrix}, \qquad \Sigma \in \mathbb{R}^{(3+2M) \times (3+2M)}$$

The key structural fact is that $\Sigma$ is dense: observing a landmark from a new pose correlates the landmark estimate with the pose estimate and, through the pose, with every other landmark. This density is not an accident or an implementation nuisance; it is the mathematical expression of the chicken-and-egg coupling. If the pose is uncertain, then a landmark's inferred location inherits that uncertainty, and so does every other landmark placed relative to the same pose. Each measurement update tightens the whole web. Algorithmically, each step is the familiar predict-then-update cycle, with the motion model linearized in the pose block and the measurement model (range and bearing to a landmark) linearized jointly in the pose and landmark blocks. Linearization is needed because both models are nonlinear in the pose, through sines and cosines of the heading and through the arctangent in the bearing, and the EKF evaluates the local Jacobian at the current mean. As everywhere with extended Kalman filtering, the linearization is only trustworthy within a neighborhood, so large innovations or long un-observed stretches can push the estimate to where the linearization misleads.

Let's implement a compact EKF-SLAM. To keep the demo focused on the estimation machinery, we make three explicit simplifications: the robot follows a known circular path with noisy odometry, measurements are range-bearing to landmarks with known data association (we are told which landmark each measurement corresponds to; the real problem of deciding this is the association problem), and landmark initialization uses the first measurement directly. Each simplification hides a real difficulty. The association problem, deciding whether this observation is of a previously mapped landmark or a new one, is a discrete inference problem layered on top of the continuous one, and a single wrong association can corrupt the entire dense covariance. Initialization from a single range-bearing reading is possible only because the sensor directly measures both range and angle; with a camera, a first sighting constrains only a ray, and the landmark's depth stays unknown until parallax accumulates. The time schedule also matters: the filter consumes the measurements at $t=0$ without advancing the state, then performs one odometry prediction from $t-1$ to $t$ before applying the measurements indexed by $t$. Thus every stored estimate, measurement set, and ground-truth pose with the same index refers to the same simulation time. Because the demo has only relative range-bearing observations and unknown landmarks, its global $SE(2)$ frame is not identifiable from the measurements. The imperfect initial pose prior supplies the simulator-frame anchor, so the raw pose and map errors below are unaligned and depend on that gauge choice.

Recognition-based loop closure is itself an inference problem, often approached with learned or hand-crafted place descriptors followed by geometric or probabilistic verification. False loop closures are especially dangerous because a wrong long-range constraint can warp an otherwise consistent map.

Landmarks are sparse. For planning, agents often want a dense answer to a dense question: for each cell of space, what is the probability it is occupied? Collision checking and path planning must reason about every point the robot might sweep through; detected features alone are insufficient. A landmark map tells you where distinctive things are; an occupancy map estimates where space is free, occupied, or unknown and provides a key input to collision checking and path planning. Occupancy grids discretize space into cells and maintain, per cell, a Bernoulli belief. Two strong assumptions concern the map itself, and it is important to state them:

• Cell independence: the map posterior is approximated as factorizing across cells conditional on the pose and measurement history. This is clearly false at boundaries, where adjacent cells belong to the same surface, but it buys an enormous simplification: instead of one inference problem over the joint map, we get a grid of tiny independent problems.
• Static world: occupancy does not change over time, so evidence can be accumulated indefinitely. This is what allows the filter to integrate over the whole history, and it is also what breaks when furniture moves.

The standard recursive update also conditions on a known or estimated sensor pose and treats measurements as conditionally independent given the map, pose, and inverse sensor model. Correlated rays or pose errors violate that approximation and can make repeated evidence accumulate too confidently.

Under these map and measurement assumptions, each cell admits a small binary filter, and the standard trick is to run it in log-odds form. For occupancy probability $p$, define $l = \log \frac{p}{1-p}$. Bayes updates multiply odds, which become additions in log space; the update for cell $i$ given observation $\mathbf{z}$ is:

$$l_i \leftarrow l_i + \log \frac{p(m_i \mid \mathbf{z}_t)}{p(\lnot m_i \mid \mathbf{z}_t)} - \log \frac{p(m_i)}{p(\lnot m_i)}$$

The first term is the inverse sensor model: given "this ray passed through here and stopped there," how much does that raise or lower each cell's occupancy? Rays mark traversed cells as likely free and their endpoint as likely occupied. Everything before a valid endpoint is evidence against occupancy because the measurement indicates that the beam traversed those cells before returning from the endpoint. The endpoint is evidence for occupancy because something returned or stopped the beam there. The subtraction of the prior log-odds prevents repeatedly re-adding prior belief on every update; without it, a cell would drift toward the prior's preference every time any ray touched it, even a completely uninformative one. Clipping keeps probabilities from saturating at 0 or 1, preserving the ability to revise beliefs later. This last detail is easy to dismiss and important to keep: a cell that has saturated at "certainly free" cannot absorb future evidence that it is occupied, and in any environment where the map might be wrong or the world might change, certainty is a liability.

The deeper lesson for world-model builders is what this filter does with negative evidence. A cell being consistently observed as free is information, just as real as an obstacle. The ray that traveled through a cell and kept going has testified that nothing solid was there, and after enough such rays the cell's probability drops far below the prior. Learned world models must therefore be evaluated for whether they distinguish unknown space from sensor-supported free space. Occupancy grids are a reminder that an observed ray traversal is evidence of free space under the sensor model. Mere absence of a measurement is not automatically free-space evidence, because it may reflect occlusion, field of view, dropout, or missing coverage. A state representation that cannot distinguish "unknown" from "known empty" will mislead any planner that relies on it.

Let's build a small occupancy map from simulated lidar.

The reconstructed map records occupied boundary cells for all three obstacles, but notice the structure of what it does not know: obstacle interiors, cells behind obstacles, and regions no ray ever entered sit at or near the prior probability 0.5 unless another ray supplies evidence. This shadowing is not a bug. It is the honest representation of perspective dependence, and it is precisely what a naive "render the world from the last frame" world model loses. A rendered frame shows surfaces that happened to be visible; it has no way to say that the region behind the pillar is unverified, and a planner reading it has no way to ask. This distinction supports information-aware planning: a planner can prefer to gather information before committing when its objective explicitly penalizes uncertainty or rewards information gain. This is an early instance of a theme that will recur throughout the book: a state representation with honest uncertainty does not just support estimation, it actively shapes behavior, driving exploration and information gathering as rational responses to what the state admits it does not know.

Occupancy answers "is something here?" Most tasks also need "what is here?" A robot that knows a region is occupied but not whether it is a table, a person, or a pile of boxes can still choose a conservative detour or gather more information. Occupancy alone cannot choose well among all task-specific responses, such as whether waiting or interaction is appropriate. Semantic maps attach class distributions to geometric elements, either per cell (Bayesian kernel semantic fusion: each labeled observation casts a kernel of evidence into the grid) or per object in a landmark-style map. The per-cell version treats every classified pixel or point as a noisy categorical observation of the cell it lands in, and fuses those observations with the same Bayesian accumulation machinery as occupancy; the per-object version instead maintains discrete object hypotheses with class posteriors, which is more compact and more natural for manipulation but requires solving the object segmentation and data association problems first. A prominent representation in autonomous-driving research is 3D semantic occupancy: a volumetric grid where every cell carries both occupancy and class, effectively fusing detection and mapping into one representation. Its appeal is that it answers, in a single queryable structure, both the geometric question a planner asks and the semantic question a policy asks, without requiring objects to have been detected as objects at all. Semantic maps introduce new failure modes that pure geometry avoids: class priors can override evidence (the map "knows" tables do not float), and semantic errors accumulate through categorical, often correlated likelihoods rather than exactly like geometric errors. Multi-view semantic fusion can improve a class estimate, but it lacks the same direct metric constraint that triangulation provides for depth. Two cameras seeing the same surface from different angles can cross-validate a depth estimate; classifiers seeing the same object from different angles may share biases and agree on a wrong label with high confidence. Semantic disagreement is therefore often harder to diagnose than geometric inconsistency.

Dense uniform grids are simple but carry the discretization with them: resolution is fixed, memory grows with the bounded volume, and smooth surfaces become staircases. Each of these costs bites in a different way. Fixed resolution means choosing, at map-building time, a level of detail that may be wrong for every downstream use; memory growing with the full volume means that the empty air in a large room costs as much as the furniture; and staircase artifacts mean that surface normals, which contact planners and physics engines need, are poorly defined at exactly the boundaries that matter. Sparse voxel, octree, and multiresolution grids alter these trade-offs by allocating storage selectively. A modern alternative is to represent the scene as a continuous function of coordinates, parameterized by a neural network.

A neural field is a learned mapping $f_\theta: \mathbf{x} \mapsto \mathbf{y}$ from spatial coordinates to scene quantities: occupancy probability, signed distance, density, color, or radiance. Querying it anywhere in continuous space evaluates the representation rather than indexing a fixed output voxel. This inverts the usual relationship between representation and query: a grid stores answers at grid points and interpolates between them, while a field computes the answer on demand. For a pure fixed-size coordinate MLP, parameter storage does not grow with the queried spatial volume, although resolution remains limited by model capacity. Hybrid neural fields can add multiresolution hash grids, sparse voxels, or explicit primitives whose storage does scale with coverage and detail. Gradients with respect to coordinates are available by automatic differentiation, which matters for optimization and analysis. The components are:

• Coordinate input: a position, optionally with viewing direction or time.
• Feature mapping: an encoding of coordinates (more on this below) that makes the target function learnable.
• Field network: a small MLP mapping encoded coordinates to scene quantities.
• Query-based supervision: the network is trained on observations tied to coordinates, such as "this ray terminates here" or "this pixel has this color."

The immediate question is why an MLP is a good idea at all, because vanilla coordinate networks have a well-documented weakness: ReLU MLPs are piecewise linear and often learn low-frequency structure before high-frequency detail, a phenomenon called spectral bias. Under a fixed model and optimization budget, that preference can blur the sharp boundaries that geometry depends on. A standard remedy is a Fourier feature mapping: before feeding coordinates to the network, map them through random frequencies,

$$\gamma(\mathbf{x}) = \left[\cos(2\pi B \mathbf{x}), \; \sin(2\pi B \mathbf{x})\right]$$

with $B$ a matrix of random frequencies. The sinusoidal encoding exposes basis functions at selected spatial frequencies and changes the optimization bandwidth. The MLP can combine those features to fit fine detail instead of constructing all high-frequency variation from many raw-coordinate piecewise-linear regions. The frequency distribution in $B$ is a design parameter: too narrow a bandwidth loses detail, while an unnecessarily wide bandwidth can increase noise-fitting risk with finite samples and insufficient regularization. Bandwidth and regularization should be tuned against held-out data. Let's see the effect directly, by fitting a 2D occupancy field (a toy stand-in for a 3D neural field) with and without Fourier features.

One of the most influential neural-field formulations for view synthesis is the neural radiance field (NeRF). Here $f_\theta$ maps a 3D position and viewing direction to emitted color and volume density. The viewing-direction input is what allows the field to represent view-dependent appearance, such as specular highlights that move as the camera moves, while the position input carries the underlying geometry. A camera image is rendered by volume rendering: march a ray through the field, and accumulate color weighted by transmittance,

$$C(\mathbf{r}) = \sum_{i=1}^{N} T_i \, \alpha_i \, \mathbf{c}_i + T_{N+1}\mathbf{c}_{\mathrm{bg}}, \qquad T_i = \prod_{j < i} (1 - \alpha_j), \qquad T_{N+1} = \prod_{j \le N} (1 - \alpha_j)$$

Here $N$ is the number of ordered ray intervals, $\mathbf{c}_i$ is the color predicted at sample $i$, and $\mathbf{c}_{\mathrm{bg}}$ is the background color. Each $\alpha_i \in [0,1]$ is the conditional opacity, or termination probability at sample $i$ given that the ray reached that interval, and $T_i$ is the probability it survived all earlier intervals. Their product $T_i\alpha_i$ is the unconditional probability mass assigned to termination at sample $i$. The residual survival mass $T_{N+1}$ weights the background color $\mathbf{c}_{\mathrm{bg}}$; that term vanishes for a black background and becomes negligible when the sampled volume is effectively opaque. The formula is a soft, differentiable version of ray casting: instead of taking the first surface hit, it sums contributions from all samples along the ray, each discounted by the probability that the ray was already absorbed before reaching it. Samples deep behind an opaque surface contribute almost nothing because $T_i$ has decayed to nearly zero, which is how occlusion emerges from a fully differentiable sum. Because rendering is differentiable, the network trains directly from posed images by gradient descent on photometric reconstruction. A shared radiance field couples the training views through one differentiable rendering model. Standard NeRF assumes known camera poses and jointly optimizes density and appearance; later methods such as BARF additionally parameterize and refine camera poses. This shared rendering objective encourages cross-view consistency, but photometric ambiguity, calibration error, and optimization pathologies mean it does not guarantee unique or correct geometry.

Three practical lineages matter for world models:

• Fast per-scene fields: multiresolution hash encodings and sparse explicit structures can reduce per-scene optimization dramatically by shifting work toward local learned lookups. The achieved runtime depends on the method, hardware, scene, optimization budget, and target quality. These approaches replace much of a deep MLP's work with learned values on a multiresolution structure, so the network refines locally stored detail. Speed matters for world models because a representation that is too slow to update cannot participate in the online perception-prediction-action loop.
• Feed-forward generalization: rather than optimizing per scene, networks predict field parameters (or directly predict geometry) from one or a few images, amortizing a learned prior over scenes. This is the geometry-flavored cousin of the amortized inference we discussed in Part III. The optimization-based version finds the scene that best explains the images; the feed-forward version predicts a scene from learned priors in one pass, and some systems optionally refine that prediction with per-scene optimization. Feed-forward inference is faster but prior-dependent in the same broad sense as monocular depth.
• Explicit hybrid representations: 3D Gaussian splatting represents the scene as a set of anisotropic 3D Gaussians with positions, covariances, opacities, and colors, optimized through a differentiable rasterizer. It keeps the differentiable, multi-view-consistent training recipe of NeRF but replaces the implicit field with an explicit, sortable, splattable primitive set, which is what makes real-time rendering possible. The explicit form exposes direct per-primitive operations: primitives can be added, removed, moved, and grouped. Later systems can exploit those operations for editing or dynamics, but object-level use still requires correspondence, persistent identity, and grouping mechanisms.

Real environments change. Extending neural fields to 4D means representing $f_\theta(\mathbf{x}, t)$, and the interesting design question is how time enters, because different answers buy different structure. Three prominent families are useful for organizing the literature:

• Time-conditioned fields: add $t$ (or a time encoding) as network input. This simple parameter sharing can encourage smoothness, but it provides no explicit temporal-coherence guarantee and may generalize poorly beyond observed times. That limitation motivates structured deformation or motion models for world-modeling tasks.
• Canonical space plus deformation: model a static canonical scene and a learned warp $\mathbf{w}(\mathbf{x}, t)$ that maps time-$t$ points into canonical space. Dynamics become a flow of geometry rather than an independent stack of frames, which regularizes the model and makes motion an explicit, queryable quantity, closely related to the scene flow of the previous section. The structural assumption is that most of the scene persists and only moves: appearance and shape live in the canonical field, and the warp carries the dynamics. When that persistence assumption fails, a single topology-preserving deformation of a fixed canonical field cannot represent births, deaths, or topology changes without additional time-dependent density, visibility, appearance, or residual mechanisms. More general non-injective warps or time-varying field support relax this limitation but also weaken the simple persistent-scene interpretation.
• Per-time explicit representations: sequences of Gaussian splats or meshes with motion fields between them form an active family in dynamic-scene and driving-scene reconstruction. This family gives up some of the compactness of a single global representation in exchange for per-frame flexibility, and the motion fields between frames reconnect the frames so that motion remains a first-class quantity rather than an artifact of frame-to-frame comparison.