Observations, Sensors, and Multimodal Data › Summary

• Observations are projections, not states. Physical observations may be lossy, delayed, noisy, or imperfectly calibrated, and downstream models inherit the defects that are present. The effective observation pipeline and its noise must be characterized and maintained.
• Modalities have complementary failure modes. Joint and inertial signals offer frequent body measurements; vision supplies rich spatial evidence but suffers occlusion and latency; touch covers the contact interface; audio can cue events outside the camera view; language carries asynchronous symbolic information.
• Calibration and synchronization make streams commensurable. Extrinsics and intrinsics put measurements in a shared space; hardware triggers, clock discipline, and honest timestamps put them on a shared timeline. The timestamp is part of the measurement. Calibration and time-offset estimation are coupled problems, both drift over time, and both fail in ways that masquerade as model failures rather than data failures.
• Missing, delayed, and noisy observations have principled treatments. Missing data means no update, so uncertainty grows; delays are handled by buffer-and-refilter, state augmentation, or forward prediction; noise requires modeling its true shape, including bias and drift, not just assuming Gaussians. All of the treatments reuse belief-state machinery from Part II, and state augmentation, promoting nuisance quantities like bias or delayed states into the estimated state, is the recurring trick.
• Fusion benefits from independent evidence and requires alignment. For two finite-variance independent Gaussian measurements, precision weighting reduces variance below either input. Shared errors reduce that gain, and temporal, spatial, and semantic relationships must be modeled correctly.
• Classical and learned fusion are complementary. Filters expose their calibration and noise assumptions; learned fusion can model relationships that are difficult to write down but may absorb dataset artifacts. Many systems combine the two.

With a clear picture of what our sensors give us, we can now ask the sharper question that opens the next chapter: what must a representation contain for a world model to be correct, that is, what makes a state sufficient?