Observations, Sensors, and Multimodal Data › Observations, Sensors, and Multimodal Data

Imagine a small quadruped robot walking across a cluttered laboratory. In this illustrative setup, its cameras deliver 30 frames per second, each frame already 40 milliseconds old by the time it reaches the onboard computer. Its joint encoders report angles 500 times per second with negligible processing latency, but gearbox backlash creates direction-dependent position error. An IMU jitters at 1 kHz. A thresholded contact event from each foot is available only after force crosses its detection threshold; depending on sensor latency and gait timing, the controller may already have advanced to another phase. None of these signals is the robot's state. None of them is even a complete description of a single instant in time. And yet the robot's world model, its planner, and its gait controller all have to act as if they were.

Notice how each of these failure modes is different in kind, not just in degree. The camera's problem is delay: the information is accurate but stale. The encoder's problem is hysteresis: its error depends on the direction from which the joint approached the reading. The IMU's problem is noise: integrating accelerometer or gyroscope noise and bias accumulates velocity, position, or orientation uncertainty unless filtering and external corrections constrain it. The foot sensor's problem is event structure: a thresholded report follows the physical onset of contact. A single "sensor reading" abstraction hides all of these distinctions, and systems that hide them pay for it later.

This chapter is about the raw material every world model consumes: observations. In Part II we learned how to filter, identify, and control dynamical systems given signals, but we mostly treated those signals as given. Here we open the box. What does each physical sensor measure? Why do streams disagree about timing, space, and noise? How do we calibrate and synchronize them? And how do we fuse them into a coherent belief that a downstream world model can consume?

The answers to these questions are not incidental engineering details that can be deferred until after the interesting modeling work is done. They constrain everything downstream. The central discipline of this chapter can be stated in one sentence: a deployed physical observation is often a lossy, delayed, noisy, or imperfectly calibrated projection of state, and downstream models inherit whichever defects are present. A world model trained on badly aligned sensor streams may learn the misalignment. A planner that ignores sensor latency acts on an estimate of the past. A policy that overfits to one modality may fail when that modality drops out. In an idealized simulation configured with instantaneous, noiseless, perfectly registered observations, none of these problems appears unless the designer adds them. Unmodeled sensing defects are one contributor to sim-to-real failure, alongside mismatches in physics, actuation, rendering, and environment variation.