Observations, Sensors, and Multimodal Data

Imagine a small quadruped robot walking across a cluttered laboratory. In this illustrative setup, its cameras deliver 30 frames per second, each frame already 40 milliseconds old by the time it reaches the onboard computer. Its joint encoders report angles 500 times per second with negligible processing latency, but gearbox backlash creates direction-dependent position error. An IMU jitters at 1 kHz. A thresholded contact event from each foot is available only after force crosses its detection threshold; depending on sensor latency and gait timing, the controller may already have advanced to another phase. None of these signals is the robot's state. None of them is even a complete description of a single instant in time. And yet the robot's world model, its planner, and its gait controller all have to act as if they were.

Notice how each of these failure modes is different in kind, not just in degree. The camera's problem is delay: the information is accurate but stale. The encoder's problem is hysteresis: its error depends on the direction from which the joint approached the reading. The IMU's problem is noise: integrating accelerometer or gyroscope noise and bias accumulates velocity, position, or orientation uncertainty unless filtering and external corrections constrain it. The foot sensor's problem is event structure: a thresholded report follows the physical onset of contact. A single "sensor reading" abstraction hides all of these distinctions, and systems that hide them pay for it later.

This chapter is about the raw material every world model consumes: observations. In Part II we learned how to filter, identify, and control dynamical systems given signals, but we mostly treated those signals as given. Here we open the box. What does each physical sensor measure? Why do streams disagree about timing, space, and noise? How do we calibrate and synchronize them? And how do we fuse them into a coherent belief that a downstream world model can consume?

The answers to these questions are not incidental engineering details that can be deferred until after the interesting modeling work is done. They constrain everything downstream. The central discipline of this chapter can be stated in one sentence: a deployed physical observation is often a lossy, delayed, noisy, or imperfectly calibrated projection of state, and downstream models inherit whichever defects are present. A world model trained on badly aligned sensor streams may learn the misalignment. A planner that ignores sensor latency acts on an estimate of the past. A policy that overfits to one modality may fail when that modality drops out. In an idealized simulation configured with instantaneous, noiseless, perfectly registered observations, none of these problems appears unless the designer adds them. Unmodeled sensing defects are one contributor to sim-to-real failure, alongside mismatches in physics, actuation, rendering, and environment variation.

Building on the POMDP formulation from Part II, recall that an agent never observes the environment state $s_t$ directly. Instead it receives an observation through an observation function:

The observation function maps a true state to the sensor report, with a noise term capturing measurement corruption. Formally, model the observation at time $t$ as:

where:

• $o_t$: the observation the agent receives at time $t$
• $s_t$: the true environment state at time $t$, which the agent never sees directly
• $h$: the observation function, the (possibly stochastic) measurement model of the sensor
• $\eta_t$: auxiliary nuisance variables drawn from $p_\eta$, such as random sensor noise or a latent occlusion state. Deterministic effects such as quantization and saturation belong in the possibly nonlinear map $h$. When we worked with Kalman filters and belief states in the Bayesian filtering chapter, we assumed we knew $h$ and the statistics of $\eta_t$. In real systems, both must be earned through calibration, characterization, and careful data engineering. That engineering is the subject of this chapter.

The innocent symbol $h$ packs in a great deal. In the cleanest textbook setting, $h$ is a fixed, known function and $\eta_t$ is an independent, identically distributed Gaussian. In a real robot, the effective observation pipeline includes the lens distortion of the camera, the mounting angle of the sensor bracket (which changes when the robot is bumped), the gain of the amplifier, the compression algorithm applied to the image before it reaches memory, and the encoder network that turns pixels into features. Timestamping and delay add a temporal relationship that may be modeled inside an augmented observation model or tracked separately. Some components are stable and calibratable; others drift with temperature or vary between units of the same sensor model. The techniques in this chapter either shrink the gap between the textbook model and the deployed pipeline or represent it explicitly.

This three-level vocabulary matters because design decisions live at each level. Latency compensation is an observation-level concern: a single delayed frame can be rewound and refiltered. Redundancy is a sensor-level concern: two cameras can be placed so that one's blind spot is the other's field of view. Architecture is a modality-level concern: an image encoder differs from a scalar encoder for joint angles because their representational formats differ. Keeping the levels distinct prevents a problem caused by one bad sensor from being misdiagnosed as a property of the whole modality.

Three properties of the observation map deserve emphasis, because each one causes a distinct class of failure downstream:

• Lossiness. Many states map to the same observation. A camera sees the front of an object but not its back; a joint encoder reports angle but not torque. The observation function is many-to-one, so the agent must maintain a belief over states, not a point estimate. Lossiness is not a defect to be engineered away; it is a consequence of physics and information theory. The design question is whether the task-relevant aspects of the state survive the projection. A gripper camera that cannot see whether an object is slippery has lost information the manipulation policy needs, and no amount of downstream cleverness will recover it from that stream alone.
• Delay. Sensing takes time: photons integrate, signals travel, buses serialize, software queues. An observation that arrives at time $t$ usually describes the world at some earlier time $t - \Delta$. We call $\Delta$ the sensor latency. Delay is insidious because it is invisible in the data itself: a delayed stream looks statistically identical to an on-time one, and only comparison against ground truth, or against an independent fast sensor, exposes it. For slow-changing environments delay is harmless; for fast dynamics it converts every estimate into a statement about the past.
• Noise and bias. Measurement error is rarely a clean zero-mean Gaussian. Real sensors exhibit bias, drift, heavy-tailed outliers, and noise that depends on the signal itself (for example, visual noise that grows in low light). The distinction between noise and bias is worth internalizing early: noise averages out given enough repeated measurements, while bias does not, no matter how long you average. A sensor with high noise but zero bias can be rescued by fusion and filtering; a sensor with low noise but a large bias will confidently drag every fused estimate toward the wrong answer.

Why does this matter for world modeling specifically? A predictor trained on observations inherits the observation function. If its "state" is a delayed camera image, the learned dynamics model predicts the world of 40 milliseconds ago, and plans are evaluated against a world that has moved on. Separating prediction from use requires an exact account of the observations. This is also the first place where we must distinguish four notions of model quality: predictive fidelity, representation quality, uncertainty quality, and decision usefulness. Strong next-frame prediction on a delayed stream can coexist with poor control performance. Prediction loss alone cannot tell you which quality the model achieved.

Different sensors occupy different points in the tradeoff space of bandwidth, latency, noise, coverage, and physical risk. No single sensor wins on all axes. Before we can fuse modalities, we need a concrete sense of what each one offers. Table modality-properties gives illustrative device-level ranges for five modalities used in robotics and embodied AI; actual values depend on the device, configuration, preprocessing pipeline, and definition of an observation update.

Read the rate column against the latency column and a useful engineering tendency appears. Many stacks pair high-rate, low-dimensional inertial or joint signals with slower, higher-dimensional perceptual streams. A kilohertz IMU sample is a handful of numbers; a 30 Hz image may contain millions of pixels whose task meaning requires substantial processing to extract. Audio is a reminder not to turn this tendency into a law: its waveform carrier is sampled quickly even when the downstream event or feature updates are much slower. These differences are one practical reason layered estimation stacks use fast low-level loops and slower high-level perception.

The contrast can be sketched on a scatter plot, with each modality's characteristic raw sampling rate against a rough description of one sample. We encode richness as an illustrative ordinal score from 1 (a few scalars) to 5 (high-dimensional or symbolic content). The score is not a physical quantity, and the axes mix carrier sampling with event-driven updates, so the figure is a design sketch rather than a measured law:

The illustrative points show a broad tendency, not a per-decade rule. Proprioceptive and tactile streams offer frequent compact updates; vision and language offer richer but slower observations; audio sits apart because raw waveform rate and semantic event rate are different quantities. A layered stack is one effective response to those mismatched rates and representations, but the figure does not establish a universal sensor law.

RGB and depth cameras often produce high-data-rate observations with rich spatial and semantic cues. A single RGB frame can contain evidence about object identity, layout, texture, and lighting; a calibrated depth camera or stereo pair adds metric geometry. This richness is why much of the world-model literature, which we will see in Part V, predicts future frames. Generalizing that prediction across varied scenes benefits from information about object persistence, motion, illumination, and geometry. The target encourages representations of predictive dynamics, but it does not prevent memorization or shortcut correlations by itself.

The costs are equally important. Conventional passive cameras depend on ambient light, suffer from motion blur when either the camera or the scene moves, and provide no direct measurement of surfaces they cannot see. Occlusion means the effective observation rate for any particular object can be far lower than the frame rate. A camera producing 30 frames per second is, for an object hidden behind a shelf, producing no new visual information about that object. Vision pipelines also tend to pass through a learned encoder before downstream models see them. That encoder is part of the effective observation function. Training it on a different distribution or fine-tuning it mid-project changes that function, so models trained before the change no longer consume the representation on which they were calibrated. This can be a subtle source of irreproducibility.

Microphones sample fast, consume little power, and can detect sound around obstacles. For a robot, audio often acts as an event cue: a dropped tool, a change in motor pitch, footsteps in the next room. Among the local physical modalities in this tour, it is especially useful for audible events outside the camera's field of view because sound diffracts and reflects. Its spatial precision is usually poorer than vision, and reflections complicate timing-based localization indoors. With a clear line of sight, the direct wave ordinarily arrives before reflections. When the direct path is blocked or severely attenuated, however, the earliest detectable wavefront may be diffracted or reflected, so a naive time-of-arrival method can infer the wrong direction. Audio can therefore provide an early cue that vision then examines in detail.

Language differs from direct physical sensing. It is an asynchronous symbolic observation or conditioning channel carrying task specifications, corrections, and attributed knowledge such as "the red cup is fragile." Appearance may suggest a material property, but language can provide attributes that vision alone cannot establish reliably. In a world model, language may shift priors over dynamics and goals rather than measure an instantaneous physical state. Told that the cup is fragile, the model receives a constraint on acceptable future trajectories and relevant physical parameters such as contact tolerance. A message can still be represented as a POMDP observation, but treating it as a synchronized measurement of the current scene would discard its loose time indexing and underspecification.

Tactile sensors directly measure pressure, shear, vibration, and temperature at the contact interface. In many gripper configurations, the robot body partially or completely occludes that interface from ordinary external vision, so touch supplies information the vision system may miss. Tactile arrays also wear, and their calibration can drift with temperature and use. High-rate tactile signals can capture the early stages of contact formation and slip. Slip may unfold over milliseconds; on platforms whose vision loop is slower than the contact dynamics, touch can provide the local feedback needed for reactive control. High-speed or event-based vision can also contribute when the platform supports it.

Proprioception is the agent's sense of its own body: joint angles from encoders and body-motion signals from IMUs. Motor currents and internal load signals are often grouped as interoception. These measurements are fast and usually available, which is why high-bandwidth inner control loops rely heavily on them. Outer-loop estimation and control often add exteroceptive aids such as vision, range sensing, or GPS. Proprioceptive weaknesses include drift and bias: wheel odometry accumulates unbounded error, IMU orientation estimates wander without an external reference, and gear backlash introduces direction-dependent error that no constant offset can correct. An unaided odometry estimate does not merely get noisier over time; its absolute error can grow without bound, and that drift cannot be identified from the dead-reckoning stream alone. A landmark, vision measurement, or GPS fix can re-anchor it.

The distinction has practical consequences beyond bookkeeping. For direct joint encoders, the observation map can be close to the identity. Other body sensors are less direct: IMU readings depend on frame orientation, gravity, dynamics, bias, and calibration. Exteroceptive signals depend on the relative pose between sensor and environment, so a camera reading depends jointly on where an object is and where the sensor points. Some calibration problems use known agent kinematics, as hand-eye calibration does; others estimate camera or target motion directly from calibration observations. When an estimator does rely on agent kinematics, errors in the self-model can masquerade as errors in the world model.

One more observation ties this section together: proprioception is often fast but local to the body, while exteroception can be richer but slower or intermittent. Bayesian filtering is designed for that complementary structure. Its prediction step propagates belief through the dynamics, while updates incorporate measurements at their actual rates. Each modality retains different failure modes. The worked example is configured so a frequent noisy encoder and a slower, more accurate camera improve one another when their uncertainties and timestamps are modeled correctly.

Before any fusion or learning can happen, the spatial frames and measurement times of different sensors must be related through known or estimated transforms and clock offsets. An implementation may transform data into one common frame and clock, or retain native coordinates and apply those relationships inside the estimator. Calibration handles the spatial relationship; synchronization handles the temporal one. Errors in either propagate into downstream estimates and can be mistaken for model error.

Each sensor has its own coordinate frame: the camera's optical axis, the IMU's case, the tactile array's surface. The rigid-body transforms between these frames and the robot body are extrinsics. A sensor's intrinsics describe its internal measurement geometry; for a camera, they include focal length, principal point, and lens distortion. Camera intrinsics alone do not map the world into pixels: that projection combines intrinsics with the camera pose supplied by extrinsics. Associating raw camera and lidar measurements therefore requires the relevant intrinsics plus the transform between their frames. If upstream processing has already produced metric quantities in calibrated frames, downstream fusion may consume those quantities without applying every raw calibration parameter again.

For cameras, intrinsics come from standard calibration procedures using a known pattern viewed from many poses. The known geometry of the pattern provides correspondences between world points and image points, and the calibration solves for the parameters that best explain all the correspondences at once. Extrinsics between a camera and the robot's hand or base come from hand-eye calibration: move the robot to many poses, observe a fixed target from each, and solve for the unknown transform from the resulting loop-closure equations. The name comes from the classic robot-arm setting, where the "eye" is a camera in the hand, but the technique applies to any pair of one moving sensor and one moving body. The reason many poses are needed is that each pose contributes one loop-closure equation, and a single equation cannot disentangle the unknown camera-to-hand transform from the unknown target-to-world transform; only the accumulated constraints across many motions pin both down. IMU-to-camera extrinsics are estimated similarly, often jointly with time offset in unified calibration toolkits, because spatial and temporal calibration turn out to be coupled: an unknown time offset between a camera and an IMU produces errors that mimic a spatial misalignment, and vice versa.

Two practical warnings. First, calibration is not a one-time event: thermal expansion, impacts, and mounting creep can change extrinsics. Production systems should recalibrate or monitor residuals, and some perform that monitoring online. A robot calibrated in the morning and dropped at noon may no longer satisfy its calibration tolerance. Second, a wrong extrinsic can look like a dynamics error. A camera mounted one centimeter from its assumed pose can make a forward model predict systematically displaced objects. Before retraining the model, audit the transforms that produced its data.

Timing is the more insidious problem, because there is no visible artifact when it goes wrong. A spatial misalignment at least produces geometrically impossible overlays that a human can spot in a visualization; a temporal misalignment produces data that looks entirely plausible and is simply wrong. Three distinct time intervals get confused:

• Sampling time: the instant the physical measurement was taken (the midpoint of the camera's exposure, for instance).
• Timestamp: the time attached to the observation by software, which may be assigned at readout, at driver receipt, or at consumer arrival.
• Arrival time: when the downstream component receives the data.

Ideally the timestamp represents sampling time on a common clock, while arrival time and end-to-end latency are recorded separately. In practice, timestamps are assigned late, clocks on different devices can drift apart by milliseconds per minute, and queues add jitter. The gap between sampling time and timestamp can exceed the latency budget of a fast control loop and vary with bus contention or driver behavior. The timestamp is itself an estimate of when the measurement occurred, produced by an imperfect clock.

The mitigations form a hierarchy:

• Hardware triggering uses a physical sync line or common strobe to lock acquisition instants across devices. Clock synchronization, including Precision Time Protocol (PTP, standardized by IEEE 1588), aligns device clocks; devices that support scheduled acquisition can then sample against that shared timebase. These mechanisms address different parts of the timing problem and can be used together.
• Software synchronization estimates and corrects clock offsets in software, for example by cross-correlating signals that both sensors should observe, such as the IMU and camera both seeing the same acceleration spikes. The cross-correlation approach finds the lag that best aligns the two signals and treats that lag as the clock offset; it works well when the shared signal has sharp features and degrades when both streams are smooth.
• Online time-offset estimation treats the sensor offset as an unknown parameter and estimates it jointly with state, as some visual-inertial systems do. It can track offset drift continuously, at the cost of extra estimator parameters and computation.

Latency belongs to the whole pipeline. A camera's effective latency includes exposure, readout, compression, transfer, and decoding; pipelining can overlap some stages, but every stage on the critical path contributes to end-to-end delay. Control depends on the latency from photon to belief rather than the frame interval printed on the datasheet. On a platform moving at one meter per second, 80 milliseconds of unmodeled latency means the platform has traveled eight centimeters since the measurement time. Whether that is acceptable depends on the task geometry and control margin. We will handle delay explicitly in the worked example.

With calibration and synchronization in place, we can characterize three degradation modes that commonly occur in real datasets. Each has a principled treatment that traces back to the belief-state machinery from Part II.

The default assumption in filtering is additive Gaussian noise, $z_t = h(s_t) + v_t$ with $v_t \sim \mathcal{N}(0, R)$, where $z_t$ is the measurement, $s_t$ is the state, $h$ is the observation function, and $v_t$ is zero-mean Gaussian measurement noise with covariance $R$. This is often adequate, but real sensors violate it in characteristic ways. Encoder quantization error is bounded and is often approximated as uniform under suitable input or dither assumptions; without those assumptions it depends deterministically on the true value. Cameras have signal-dependent shot noise and can produce heavy-tailed outliers from reflections or sensor defects. IMUs have time-correlated errors such as random walks that an independent-noise model underestimates. When errors persist across updates, a filter that assumes independence can become overconfident.

There is also bias, a systematic offset that averaging cannot remove. Bias is especially dangerous because a model may absorb it as repeatable structure, while repeated samples do not cancel it. Large variance can be equally damaging, and noisy training data does not automatically make an ordinary model uncertainty-aware; both error types must be modeled and calibrated. Practical defenses include redundancy across sensors with independent bias sources, periodic re-zeroing against known references, and explicit bias states in the filter, which is how IMU biases are estimated in visual-inertial odometry. Instead of assuming the bias away, the filter promotes it to part of the state and estimates it alongside position. This state-augmentation trick recurs throughout estimation theory and will appear again in the next section.

Observations go missing. Cameras are occluded or blinded by glare, network streams stutter, tactile arrays fail cell by cell. Any imputation should carry an explicit mask or uncertainty rather than masquerade as a measurement. In filter terms, a missing observation means no measurement update, so uncertainty grows under prediction until the next observation arrives.

This uncertainty growth is concrete and quantitative. With the continuous white-acceleration model used later in this chapter, let $q_a$ denote the acceleration-noise spectral density. The one-step position-noise variance is $Q_{11}=q_a\,\Delta t^3/3$. Starting from zero covariance, pure prediction over $k$ steps gives $P_{11}=q_a(k\Delta t)^3/3$. A scalar position measurement with variance $R$ then reduces that prior variance to $P_{11}^{+}=P_{11}R/(P_{11}+R)$:

The cubic growth is the continuous white-acceleration signature after integrating acceleration into velocity and position. A planner can use a calibrated covariance to respond cautiously as the belief becomes stale. An unmasked imputed value would instead invite downstream code to treat a guess as a measurement.

In learning terms, masking or modality dropout can expose the model to gaps during training. Modality dropout randomly deletes whole sensor streams, encouraging less brittle dependence on any one modality. It can improve degradation behavior, but does not guarantee it when deployment failures differ from the training dropout pattern.

Missingness can also be informative. Discriminating "nothing there" from "sensor failed" is itself an inference problem, and models that track per-modality reliability can use dropout patterns as evidence. Occlusion is often spatially structured and correlated with predicted scene layout, while some failures affect whole frames or channels. These are heuristics rather than guarantees: failures can be spatially structured, and occlusion can remove an entire view, so reliability metadata and diagnostics still matter.

Delay produces the subtlest failure mode. When a measurement with timestamp $t - \Delta$ arrives at time $t$, calling it a measurement of the current state injects a systematic error proportional to how fast the world moves. Notice the structure of the error: it is not random, it is a deterministic function of the state's velocity, which means the filter cannot average it away and the resulting estimate is biased in exactly the direction of motion. In filtering language this is an out-of-sequence measurement, and there are standard remedies:

• Buffer and refilter. Keep a short history of past beliefs and controls. When the delayed measurement arrives, rewind the filter to its timestamp, apply the update, and replay the intervening predictions and updates. This is exact for linear-Gaussian models and the approach we implement below. Its costs are memory, proportional to the latency times the base rate, and compute, proportional to the same quantity per delayed measurement.
• Augment the state. Append the delayed state (or the measurement itself) to the state vector and update jointly, as delayed-state Kalman filters and visual-inertial estimators do. The state-augmentation trick from the bias discussion reappears: rather than special-casing the delayed measurement, we enlarge the estimation problem until the delayed measurement becomes an ordinary one.
• Predict forward. Apply the update at arrival time against a prediction of what the measurement would look like now, using the model to propagate from $t - \Delta$ to $t$. This often uses less history and replay work, but its accuracy depends on the transition model and on which intervening measurements it ignores.

All three remedies require a trustworthy timestamp and a dynamics model over the latency interval. Buffer-and-refilter restores a stored past belief, applies the delayed update, and then reruns the transition model forward over the latency window. Its quality therefore depends on the same transition model used elsewhere for prediction.

Given calibrated, synchronized, degradation-aware streams, the task of fusion is to combine them into a single belief or representation that is better than any individual stream. The word "better" here admits the same four-way decomposition we introduced earlier: fusion can improve predictive fidelity, improve representation quality, improve uncertainty quality, or improve decision usefulness, and these do not automatically come together. A fusion scheme that minimizes state-estimation error does not automatically produce a representation a planner can act on, and one that produces confident predictions does not automatically produce calibrated ones. Keeping the objective of fusion explicit prevents the common failure of optimizing a fusion module for a benchmark correlation while the downstream task silently degrades. Fusion strategies occupy a spectrum based on where the combination happens.

• Early fusion combines raw or lightly processed signals before substantial encoding, for example by joining pixels and joint readings near the input. It exposes low-level cross-modal interactions but requires explicit handling of heterogeneous scales, rates, and missingness. Modality-specific normalization or shallow frontends can still precede the join.
• Late fusion processes each modality independently, then combines the results, such as averaging predictors or multiplying calibrated Gaussian posteriors. It is modular and can tolerate a missing modality. It may discard interactions available only in raw or intermediate features, though a late fuser can still compare final predictions, residuals, uncertainty, and disagreement.
• Intermediate (hybrid) fusion gives each modality its own encoder, then combines the embeddings into a shared latent state with mechanisms such as attention or gating. This pattern is common in multimodal robotics because each encoder can match its input statistics while the fusion layer models cross-modal interactions.

For Gaussian linear models, fusion has an exact and instructive answer. Suppose $z_i=x+\epsilon_i$, where the two errors are independent and $\epsilon_i\sim\mathcal{N}(0,\sigma_i^2)$. Combining these two likelihoods without an additional informative prior (equivalently, using a flat prior), the fused estimate is the precision-weighted average:

where:

• $z_1$, $z_2$: the measurements reported by the two independent sensors of the same scalar quantity
• $\sigma_1^2$, $\sigma_2^2$: the noise variances of the two measurements; a noisier sensor contributes less
• $\hat{x}$: the fused estimate, which weights each measurement by the other sensor's variance (its inverse precision)
• $\sigma_{\text{fused}}^2$: the variance of the fused estimate, the harmonic-style combination of the inverse precisions

The intuition is precision weighting: the weight on $z_1$ is $\sigma_2^2 / (\sigma_1^2 + \sigma_2^2)$, so a low-variance measurement dominates. If sensor 1 is infinitely precise ($\sigma_1^2 \to 0$), the fused estimate converges to $z_1$. If the variances are equal, the estimate is the plain average and its variance is half of either individual variance. For two finite positive variances with independent errors, the fused variance is strictly smaller than either individual variance. Every Kalman update you implemented in Part II uses the same precision logic in matrix form. The independence assumption matters: two cameras looking through the same dirty protective window share part of their error, so treating them as independent double-counts information. Correlated errors often enter through shared vibration, clock error, power, or overlapping encoder training data.

The independence assumption can be made exact. For two sensors with equal variance $\sigma^2$ and error correlation $\rho$, the fused variance is $\sigma_{\text{fused}}^2 = \sigma^2 (1 + \rho) / 2$: the fusion advantage shrinks linearly to zero as $\rho$ goes to one:

At zero correlation the fused variance is half the single-sensor value. At perfect correlation the two curves meet: the second measurement adds no independent information. When a fusion system underperforms, measuring cross-modal error correlation is a useful diagnostic.

Alignment can involve three distinct registers: temporal, spatial, and semantic. A fusion design needs the subset relevant to its modalities and representation. The scalar Gaussian example needs compatible quantities and timing, while image-language fusion also needs a shared semantic representation.

• Temporal alignment: ensuring paired observations refer to the intended instants, using the synchronization machinery above. A fixed offset can be absorbed as apparent dynamics or bias when the model is not told about it. More similarly misaligned data reinforces that target convention rather than repairing the timestamps.
• Spatial alignment: expressing measurements in a common frame via calibration. Fusing a camera detection with a lidar return requires knowing where each sensor is, or the "same object" is never the same object. Spatial misalignment tends to scale with distance: a small angular calibration error produces a small offset for nearby objects and a large offset for far ones, which is one reason perception errors that look random in a table-top evaluation become systematic in large environments.
• Semantic alignment: mapping heterogeneous representations so their relationship has a learned meaning. Contrastively trained encoders, which place an image and its caption near each other in a shared embedding space, are the flagship example. In shared-embedding world models, this alignment lets a language instruction constrain a latent plan because proximity represents agreement in meaning. Other fusion interfaces, including cross-attention, gating, and learned adapters, can instead learn a cross-modal mapping without requiring raw embedding proximity.

A final framing will serve us for the rest of the book: fusion can be done as inference or as learning. The filtering view combines likelihoods using explicit noise models. A learned fuser estimates combination rules from data and can absorb biases or misalignments present in that data. The trade is partly about where assumptions live: the filter puts them in inspectable parameters such as $R$ and calibration transforms, while a learned module encodes them in weights. A common practical pattern uses classical filters for high-rate low-level state estimation and learned fusion for semantic content. A trained attention layer can learn reliability-sensitive weighting, but its coefficients are not automatically inverse variances or calibrated trust scores. The closed-form Gaussian case is a reference point, not an identity that generic attention must satisfy.

Let us build the canonical scenario from scratch. A cart moves in one dimension with position $p_t$ and velocity $v_t$, driven by acceleration commands and disturbed by random acceleration noise. This is a discretized double integrator, written as a linear state-space model:

where:

• $p_t$, $v_t$: the cart's position and velocity (the latent state; no sensor observes them exactly)
• $\Delta t$: the time step between consecutive states (0.01 s at the 100 Hz base rate below)
• $F$: the state-transition matrix, integrating velocity into position and carrying velocity forward
• $u_t$: the acceleration command applied over the interval
• $B$: the control-input matrix, mapping acceleration to changes in position and velocity
• $w_t$: the process noise, random acceleration disturbances accumulated over the step

The $\Delta t^2/2$ entry is the kinematic contribution of constant acceleration to displacement over one step. It is the discrete-time shadow of the $\tfrac{1}{2} a t^2$ term in constant-acceleration kinematics, and it exists because the acceleration acts over the whole interval, not just at its endpoint. Even though the state is two-dimensional, only the position is ever measured; the velocity must be inferred. The coupled model makes velocity observable from repeated position measurements when $\Delta t\ne 0$, whereas an uncoupled position-only model cannot infer velocity. Each position update then constrains velocity through the model, and each velocity estimate sharpens the next position prediction.

Two sensors watch it. A joint encoder measures position at 100 Hz with moderate noise, effectively instantly. A camera measures position more accurately at 10 Hz, but with 60 milliseconds of processing latency, occasional dropouts, and one long occlusion. We will build three filters, one per modality alone and one fusing both, and use a rewind-and-refilter scheme for the camera's out-of-sequence measurements. The camera contributes accurate absolute fixes; the encoder contributes frequent updates while the camera is absent.

Now we simulate the ground-truth trajectory and both sensor streams, carefully timestamping everything:

The fused estimate uses camera precision when frames arrive and encoder continuity while they do not. When a post-occlusion frame arrives, rewind applies it at generation time and replays the intervening encoder updates exactly once. This avoids injecting a stale frame as if it described the current cart position.

Now let us isolate temporal alignment from missingness. We create a dense synthetic camera stream for this diagnostic only, then fit a linear estimator of current position from the encoder at time $t$ and camera at time $t+k$. Every lag uses the same target times and the same 70/30 chronological split, so the only changed variable is the camera offset:

The minimum at zero offset confirms that the paired features should describe the same state time in this controlled setup. A 50 millisecond shift raises validation error even though the regression still fits normally. A lag sweep can therefore reveal a synchronization problem worth investigating, but a nonzero optimum is not proof by itself: the comparison must hold sample membership fixed and rule out correlated dynamics, interpolation artifacts, and clock drift.

The key parameters for the fusion example are:

• dt: Base time step (0.01 s, the 100 Hz rate at which the encoder and filter run). It appears in $F$, $B$, and $Q$, so changing it changes the dynamics, control mapping, uncertainty growth, and number of updates per physical second.
• q_accel: Continuous white-acceleration spectral density (0.09 m$^2$/s$^3$). It scales $Q$ and therefore predicted covariance growth. Larger values generally make measurement updates more influential because prediction becomes less certain.
• R_ENC: Encoder measurement noise variance (0.05$^2$ m$^2$). Smaller values make the filter trust encoder updates more heavily.
• R_CAM: Camera measurement noise variance (0.02$^2$ m$^2$). The camera is 6.25 times as precise as the encoder when a frame is available. Relative sensor variances affect their weighting, but their absolute scale relative to $Q$ and the propagated covariance also matters. Scaling only the two $R$ values changes the Kalman gain; covariance-scale invariance would require scaling $P_0$, $Q$, and all $R$ values together.
• DELAY: Camera latency in base-rate steps (6 steps, 60 ms). Replay work per delayed frame and the minimum retained buffer grow linearly with this value. This teaching implementation retains the full $T$-step state history and encoder-update array, so its actual memory use is $O(T)$; a production implementation can prune entries older than the maximum supported delay.

Careful calibration, timestamps, noise models, and fusion underpin classical spacecraft navigation and many modern drone estimators. World-model training data passes through the same kinds of sensing and alignment decisions before a network sees it. No architecture, loss, or training schedule can recover timing or geometry that the collection pipeline irreversibly discarded without additional information or assumptions.

Classical fusion requires sensor models: noise covariances $R$, calibration transforms, and latencies. In practice these are estimated, approximate, and time-varying. A wrong but positive $R$ still produces ordinary Kalman updates while silently changing the weights; innovation and consistency diagnostics can reveal the mismatch only if they are monitored. Rewind-and-refilter reproduces the exact linear-Gaussian filter when it replays all information once. An EKF or particle filter remains approximate because of linearization or finite sampling, and replaying it is computationally heavier, but rewind need not add another approximation beyond the chosen nonlinear estimator. Very long delays or high base rates also increase history storage and replay work.

The learned-fusion alternative trades explicit models for data. A cross-attention module can learn reliability-sensitive weights, but it can also exploit dataset artifacts and offers no architecture-independent guarantee under a shift in modality failures. If training always includes a functioning camera but deployment includes frequent occlusion, the learned weights may rely too heavily on vision. Likewise, a model trained on camera frames paired 50 milliseconds away from their encoder timestamps may absorb that offset unless the data or objective exposes it. A common pragmatic division of labor is classical filtering for calibrated low-level state and learned encoders or fusion for semantic content. That synthesis still depends on whether the filtered state contains what the task needs. A precise estimate of position and velocity does not tell a manipulation policy whether an object is graspable. The next chapter asks what makes a state representation sufficient.

• Observations are projections, not states. Physical observations may be lossy, delayed, noisy, or imperfectly calibrated, and downstream models inherit the defects that are present. The effective observation pipeline and its noise must be characterized and maintained.
• Modalities have complementary failure modes. Joint and inertial signals offer frequent body measurements; vision supplies rich spatial evidence but suffers occlusion and latency; touch covers the contact interface; audio can cue events outside the camera view; language carries asynchronous symbolic information.
• Calibration and synchronization make streams commensurable. Extrinsics and intrinsics put measurements in a shared space; hardware triggers, clock discipline, and honest timestamps put them on a shared timeline. The timestamp is part of the measurement. Calibration and time-offset estimation are coupled problems, both drift over time, and both fail in ways that masquerade as model failures rather than data failures.
• Missing, delayed, and noisy observations have principled treatments. Missing data means no update, so uncertainty grows; delays are handled by buffer-and-refilter, state augmentation, or forward prediction; noise requires modeling its true shape, including bias and drift, not just assuming Gaussians. All of the treatments reuse belief-state machinery from Part II, and state augmentation, promoting nuisance quantities like bias or delayed states into the estimated state, is the recurring trick.
• Fusion benefits from independent evidence and requires alignment. For two finite-variance independent Gaussian measurements, precision weighting reduces variance below either input. Shared errors reduce that gain, and temporal, spatial, and semantic relationships must be modeled correctly.
• Classical and learned fusion are complementary. Filters expose their calibration and noise assumptions; learned fusion can model relationships that are difficult to write down but may absorb dataset artifacts. Many systems combine the two.

With a clear picture of what our sensors give us, we can now ask the sharper question that opens the next chapter: what must a representation contain for a world model to be correct, that is, what makes a state sufficient?

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about observations, sensors, and multimodal data.