Geometry, Mapping, and 3D/4D State › Neural Fields and Dynamic 3D Representations

Dense uniform grids are simple but carry the discretization with them: resolution is fixed, memory grows with the bounded volume, and smooth surfaces become staircases. Each of these costs bites in a different way. Fixed resolution means choosing, at map-building time, a level of detail that may be wrong for every downstream use; memory growing with the full volume means that the empty air in a large room costs as much as the furniture; and staircase artifacts mean that surface normals, which contact planners and physics engines need, are poorly defined at exactly the boundaries that matter. Sparse voxel, octree, and multiresolution grids alter these trade-offs by allocating storage selectively. A modern alternative is to represent the scene as a continuous function of coordinates, parameterized by a neural network.

A neural field is a learned mapping $f_\theta: \mathbf{x} \mapsto \mathbf{y}$ from spatial coordinates to scene quantities: occupancy probability, signed distance, density, color, or radiance. Querying it anywhere in continuous space evaluates the representation rather than indexing a fixed output voxel. This inverts the usual relationship between representation and query: a grid stores answers at grid points and interpolates between them, while a field computes the answer on demand. For a pure fixed-size coordinate MLP, parameter storage does not grow with the queried spatial volume, although resolution remains limited by model capacity. Hybrid neural fields can add multiresolution hash grids, sparse voxels, or explicit primitives whose storage does scale with coverage and detail. Gradients with respect to coordinates are available by automatic differentiation, which matters for optimization and analysis. The components are:

• Coordinate input: a position, optionally with viewing direction or time.
• Feature mapping: an encoding of coordinates (more on this below) that makes the target function learnable.
• Field network: a small MLP mapping encoded coordinates to scene quantities.
• Query-based supervision: the network is trained on observations tied to coordinates, such as "this ray terminates here" or "this pixel has this color."

The immediate question is why an MLP is a good idea at all, because vanilla coordinate networks have a well-documented weakness: ReLU MLPs are piecewise linear and often learn low-frequency structure before high-frequency detail, a phenomenon called spectral bias. Under a fixed model and optimization budget, that preference can blur the sharp boundaries that geometry depends on. A standard remedy is a Fourier feature mapping: before feeding coordinates to the network, map them through random frequencies,

$$\gamma(\mathbf{x}) = \left[\cos(2\pi B \mathbf{x}), \; \sin(2\pi B \mathbf{x})\right]$$

with $B$ a matrix of random frequencies. The sinusoidal encoding exposes basis functions at selected spatial frequencies and changes the optimization bandwidth. The MLP can combine those features to fit fine detail instead of constructing all high-frequency variation from many raw-coordinate piecewise-linear regions. The frequency distribution in $B$ is a design parameter: too narrow a bandwidth loses detail, while an unnecessarily wide bandwidth can increase noise-fitting risk with finite samples and insufficient regularization. Bandwidth and regularization should be tuned against held-out data. Let's see the effect directly, by fitting a 2D occupancy field (a toy stand-in for a 3D neural field) with and without Fourier features.

One of the most influential neural-field formulations for view synthesis is the neural radiance field (NeRF). Here $f_\theta$ maps a 3D position and viewing direction to emitted color and volume density. The viewing-direction input is what allows the field to represent view-dependent appearance, such as specular highlights that move as the camera moves, while the position input carries the underlying geometry. A camera image is rendered by volume rendering: march a ray through the field, and accumulate color weighted by transmittance,

$$C(\mathbf{r}) = \sum_{i=1}^{N} T_i \, \alpha_i \, \mathbf{c}_i + T_{N+1}\mathbf{c}_{\mathrm{bg}}, \qquad T_i = \prod_{j < i} (1 - \alpha_j), \qquad T_{N+1} = \prod_{j \le N} (1 - \alpha_j)$$

Here $N$ is the number of ordered ray intervals, $\mathbf{c}_i$ is the color predicted at sample $i$, and $\mathbf{c}_{\mathrm{bg}}$ is the background color. Each $\alpha_i \in [0,1]$ is the conditional opacity, or termination probability at sample $i$ given that the ray reached that interval, and $T_i$ is the probability it survived all earlier intervals. Their product $T_i\alpha_i$ is the unconditional probability mass assigned to termination at sample $i$. The residual survival mass $T_{N+1}$ weights the background color $\mathbf{c}_{\mathrm{bg}}$; that term vanishes for a black background and becomes negligible when the sampled volume is effectively opaque. The formula is a soft, differentiable version of ray casting: instead of taking the first surface hit, it sums contributions from all samples along the ray, each discounted by the probability that the ray was already absorbed before reaching it. Samples deep behind an opaque surface contribute almost nothing because $T_i$ has decayed to nearly zero, which is how occlusion emerges from a fully differentiable sum. Because rendering is differentiable, the network trains directly from posed images by gradient descent on photometric reconstruction. A shared radiance field couples the training views through one differentiable rendering model. Standard NeRF assumes known camera poses and jointly optimizes density and appearance; later methods such as BARF additionally parameterize and refine camera poses. This shared rendering objective encourages cross-view consistency, but photometric ambiguity, calibration error, and optimization pathologies mean it does not guarantee unique or correct geometry.

Three practical lineages matter for world models:

• Fast per-scene fields: multiresolution hash encodings and sparse explicit structures can reduce per-scene optimization dramatically by shifting work toward local learned lookups. The achieved runtime depends on the method, hardware, scene, optimization budget, and target quality. These approaches replace much of a deep MLP's work with learned values on a multiresolution structure, so the network refines locally stored detail. Speed matters for world models because a representation that is too slow to update cannot participate in the online perception-prediction-action loop.
• Feed-forward generalization: rather than optimizing per scene, networks predict field parameters (or directly predict geometry) from one or a few images, amortizing a learned prior over scenes. This is the geometry-flavored cousin of the amortized inference we discussed in Part III. The optimization-based version finds the scene that best explains the images; the feed-forward version predicts a scene from learned priors in one pass, and some systems optionally refine that prediction with per-scene optimization. Feed-forward inference is faster but prior-dependent in the same broad sense as monocular depth.
• Explicit hybrid representations: 3D Gaussian splatting represents the scene as a set of anisotropic 3D Gaussians with positions, covariances, opacities, and colors, optimized through a differentiable rasterizer. It keeps the differentiable, multi-view-consistent training recipe of NeRF but replaces the implicit field with an explicit, sortable, splattable primitive set, which is what makes real-time rendering possible. The explicit form exposes direct per-primitive operations: primitives can be added, removed, moved, and grouped. Later systems can exploit those operations for editing or dynamics, but object-level use still requires correspondence, persistent identity, and grouping mechanisms.

Real environments change. Extending neural fields to 4D means representing $f_\theta(\mathbf{x}, t)$, and the interesting design question is how time enters, because different answers buy different structure. Three prominent families are useful for organizing the literature:

• Time-conditioned fields: add $t$ (or a time encoding) as network input. This simple parameter sharing can encourage smoothness, but it provides no explicit temporal-coherence guarantee and may generalize poorly beyond observed times. That limitation motivates structured deformation or motion models for world-modeling tasks.
• Canonical space plus deformation: model a static canonical scene and a learned warp $\mathbf{w}(\mathbf{x}, t)$ that maps time-$t$ points into canonical space. Dynamics become a flow of geometry rather than an independent stack of frames, which regularizes the model and makes motion an explicit, queryable quantity, closely related to the scene flow of the previous section. The structural assumption is that most of the scene persists and only moves: appearance and shape live in the canonical field, and the warp carries the dynamics. When that persistence assumption fails, a single topology-preserving deformation of a fixed canonical field cannot represent births, deaths, or topology changes without additional time-dependent density, visibility, appearance, or residual mechanisms. More general non-injective warps or time-varying field support relax this limitation but also weaken the simple persistent-scene interpretation.
• Per-time explicit representations: sequences of Gaussian splats or meshes with motion fields between them form an active family in dynamic-scene and driving-scene reconstruction. This family gives up some of the compactness of a single global representation in exchange for per-frame flexibility, and the motion fields between frames reconnect the frames so that motion remains a first-class quantity rather than an artifact of frame-to-frame comparison.