Observations, Sensors, and Multimodal Data › Limitations and Impact

Careful calibration, timestamps, noise models, and fusion underpin classical spacecraft navigation and many modern drone estimators. World-model training data passes through the same kinds of sensing and alignment decisions before a network sees it. No architecture, loss, or training schedule can recover timing or geometry that the collection pipeline irreversibly discarded without additional information or assumptions.

Classical fusion requires sensor models: noise covariances $R$, calibration transforms, and latencies. In practice these are estimated, approximate, and time-varying. A wrong but positive $R$ still produces ordinary Kalman updates while silently changing the weights; innovation and consistency diagnostics can reveal the mismatch only if they are monitored. Rewind-and-refilter reproduces the exact linear-Gaussian filter when it replays all information once. An EKF or particle filter remains approximate because of linearization or finite sampling, and replaying it is computationally heavier, but rewind need not add another approximation beyond the chosen nonlinear estimator. Very long delays or high base rates also increase history storage and replay work.

The learned-fusion alternative trades explicit models for data. A cross-attention module can learn reliability-sensitive weights, but it can also exploit dataset artifacts and offers no architecture-independent guarantee under a shift in modality failures. If training always includes a functioning camera but deployment includes frequent occlusion, the learned weights may rely too heavily on vision. Likewise, a model trained on camera frames paired 50 milliseconds away from their encoder timestamps may absorb that offset unless the data or objective exposes it. A common pragmatic division of labor is classical filtering for calibrated low-level state and learned encoders or fusion for semantic content. That synthesis still depends on whether the filtered state contains what the task needs. A precise estimate of position and velocity does not tell a manipulation policy whether an object is graspable. The next chapter asks what makes a state representation sufficient.