Observations, Sensors, and Multimodal Data › Missing, Delayed, and Noisy Observations

With calibration and synchronization in place, we can characterize three degradation modes that commonly occur in real datasets. Each has a principled treatment that traces back to the belief-state machinery from Part II.

The default assumption in filtering is additive Gaussian noise, $z_t = h(s_t) + v_t$ with $v_t \sim \mathcal{N}(0, R)$, where $z_t$ is the measurement, $s_t$ is the state, $h$ is the observation function, and $v_t$ is zero-mean Gaussian measurement noise with covariance $R$. This is often adequate, but real sensors violate it in characteristic ways. Encoder quantization error is bounded and is often approximated as uniform under suitable input or dither assumptions; without those assumptions it depends deterministically on the true value. Cameras have signal-dependent shot noise and can produce heavy-tailed outliers from reflections or sensor defects. IMUs have time-correlated errors such as random walks that an independent-noise model underestimates. When errors persist across updates, a filter that assumes independence can become overconfident.

There is also bias, a systematic offset that averaging cannot remove. Bias is especially dangerous because a model may absorb it as repeatable structure, while repeated samples do not cancel it. Large variance can be equally damaging, and noisy training data does not automatically make an ordinary model uncertainty-aware; both error types must be modeled and calibrated. Practical defenses include redundancy across sensors with independent bias sources, periodic re-zeroing against known references, and explicit bias states in the filter, which is how IMU biases are estimated in visual-inertial odometry. Instead of assuming the bias away, the filter promotes it to part of the state and estimates it alongside position. This state-augmentation trick recurs throughout estimation theory and will appear again in the next section.

Observations go missing. Cameras are occluded or blinded by glare, network streams stutter, tactile arrays fail cell by cell. Any imputation should carry an explicit mask or uncertainty rather than masquerade as a measurement. In filter terms, a missing observation means no measurement update, so uncertainty grows under prediction until the next observation arrives.

This uncertainty growth is concrete and quantitative. With the continuous white-acceleration model used later in this chapter, let $q_a$ denote the acceleration-noise spectral density. The one-step position-noise variance is $Q_{11}=q_a\,\Delta t^3/3$. Starting from zero covariance, pure prediction over $k$ steps gives $P_{11}=q_a(k\Delta t)^3/3$. A scalar position measurement with variance $R$ then reduces that prior variance to $P_{11}^{+}=P_{11}R/(P_{11}+R)$:

The cubic growth is the continuous white-acceleration signature after integrating acceleration into velocity and position. A planner can use a calibrated covariance to respond cautiously as the belief becomes stale. An unmasked imputed value would instead invite downstream code to treat a guess as a measurement.

In learning terms, masking or modality dropout can expose the model to gaps during training. Modality dropout randomly deletes whole sensor streams, encouraging less brittle dependence on any one modality. It can improve degradation behavior, but does not guarantee it when deployment failures differ from the training dropout pattern.

Missingness can also be informative. Discriminating "nothing there" from "sensor failed" is itself an inference problem, and models that track per-modality reliability can use dropout patterns as evidence. Occlusion is often spatially structured and correlated with predicted scene layout, while some failures affect whole frames or channels. These are heuristics rather than guarantees: failures can be spatially structured, and occlusion can remove an entire view, so reliability metadata and diagnostics still matter.

Delay produces the subtlest failure mode. When a measurement with timestamp $t - \Delta$ arrives at time $t$, calling it a measurement of the current state injects a systematic error proportional to how fast the world moves. Notice the structure of the error: it is not random, it is a deterministic function of the state's velocity, which means the filter cannot average it away and the resulting estimate is biased in exactly the direction of motion. In filtering language this is an out-of-sequence measurement, and there are standard remedies:

• Buffer and refilter. Keep a short history of past beliefs and controls. When the delayed measurement arrives, rewind the filter to its timestamp, apply the update, and replay the intervening predictions and updates. This is exact for linear-Gaussian models and the approach we implement below. Its costs are memory, proportional to the latency times the base rate, and compute, proportional to the same quantity per delayed measurement.
• Augment the state. Append the delayed state (or the measurement itself) to the state vector and update jointly, as delayed-state Kalman filters and visual-inertial estimators do. The state-augmentation trick from the bias discussion reappears: rather than special-casing the delayed measurement, we enlarge the estimation problem until the delayed measurement becomes an ordinary one.
• Predict forward. Apply the update at arrival time against a prediction of what the measurement would look like now, using the model to propagate from $t - \Delta$ to $t$. This often uses less history and replay work, but its accuracy depends on the transition model and on which intervening measurements it ignores.

All three remedies require a trustworthy timestamp and a dynamics model over the latency interval. Buffer-and-refilter restores a stored past belief, applies the delayed update, and then reruns the transition model forward over the latency window. Its quality therefore depends on the same transition model used elsewhere for prediction.