Observations, Sensors, and Multimodal Data › Observations Are Not the State

Building on the POMDP formulation from Part II, recall that an agent never observes the environment state $s_t$ directly. Instead it receives an observation through an observation function:

The observation function maps a true state to the sensor report, with a noise term capturing measurement corruption. Formally, model the observation at time $t$ as:

where:

• $o_t$: the observation the agent receives at time $t$
• $s_t$: the true environment state at time $t$, which the agent never sees directly
• $h$: the observation function, the (possibly stochastic) measurement model of the sensor
• $\eta_t$: auxiliary nuisance variables drawn from $p_\eta$, such as random sensor noise or a latent occlusion state. Deterministic effects such as quantization and saturation belong in the possibly nonlinear map $h$. When we worked with Kalman filters and belief states in the Bayesian filtering chapter, we assumed we knew $h$ and the statistics of $\eta_t$. In real systems, both must be earned through calibration, characterization, and careful data engineering. That engineering is the subject of this chapter.

The innocent symbol $h$ packs in a great deal. In the cleanest textbook setting, $h$ is a fixed, known function and $\eta_t$ is an independent, identically distributed Gaussian. In a real robot, the effective observation pipeline includes the lens distortion of the camera, the mounting angle of the sensor bracket (which changes when the robot is bumped), the gain of the amplifier, the compression algorithm applied to the image before it reaches memory, and the encoder network that turns pixels into features. Timestamping and delay add a temporal relationship that may be modeled inside an augmented observation model or tracked separately. Some components are stable and calibratable; others drift with temperature or vary between units of the same sensor model. The techniques in this chapter either shrink the gap between the textbook model and the deployed pipeline or represent it explicitly.

This three-level vocabulary matters because design decisions live at each level. Latency compensation is an observation-level concern: a single delayed frame can be rewound and refiltered. Redundancy is a sensor-level concern: two cameras can be placed so that one's blind spot is the other's field of view. Architecture is a modality-level concern: an image encoder differs from a scalar encoder for joint angles because their representational formats differ. Keeping the levels distinct prevents a problem caused by one bad sensor from being misdiagnosed as a property of the whole modality.

Three properties of the observation map deserve emphasis, because each one causes a distinct class of failure downstream:

• Lossiness. Many states map to the same observation. A camera sees the front of an object but not its back; a joint encoder reports angle but not torque. The observation function is many-to-one, so the agent must maintain a belief over states, not a point estimate. Lossiness is not a defect to be engineered away; it is a consequence of physics and information theory. The design question is whether the task-relevant aspects of the state survive the projection. A gripper camera that cannot see whether an object is slippery has lost information the manipulation policy needs, and no amount of downstream cleverness will recover it from that stream alone.
• Delay. Sensing takes time: photons integrate, signals travel, buses serialize, software queues. An observation that arrives at time $t$ usually describes the world at some earlier time $t - \Delta$. We call $\Delta$ the sensor latency. Delay is insidious because it is invisible in the data itself: a delayed stream looks statistically identical to an on-time one, and only comparison against ground truth, or against an independent fast sensor, exposes it. For slow-changing environments delay is harmless; for fast dynamics it converts every estimate into a statement about the past.
• Noise and bias. Measurement error is rarely a clean zero-mean Gaussian. Real sensors exhibit bias, drift, heavy-tailed outliers, and noise that depends on the signal itself (for example, visual noise that grows in low light). The distinction between noise and bias is worth internalizing early: noise averages out given enough repeated measurements, while bias does not, no matter how long you average. A sensor with high noise but zero bias can be rescued by fusion and filtering; a sensor with low noise but a large bias will confidently drag every fused estimate toward the wrong answer.

Why does this matter for world modeling specifically? A predictor trained on observations inherits the observation function. If its "state" is a delayed camera image, the learned dynamics model predicts the world of 40 milliseconds ago, and plans are evaluated against a world that has moved on. Separating prediction from use requires an exact account of the observations. This is also the first place where we must distinguish four notions of model quality: predictive fidelity, representation quality, uncertainty quality, and decision usefulness. Strong next-frame prediction on a delayed stream can coexist with poor control performance. Prediction loss alone cannot tell you which quality the model achieved.