Bayesian Filtering and Belief States › From Observations to Belief

Consider a drone flying indoors. Its onboard camera reports visual features, its IMU reports accelerations and rotation rates, and none of these sensors directly tells the drone where it is. Position must be inferred: the IMU integrates to a rough pose estimate that drifts, while mapped visual landmarks anchor that estimate but remain noisy and occasionally ambiguous. Neither sensor alone suffices. Their complementary failure modes make a distributional belief essential: aliasing and missed detections can preserve several plausible poses even after a landmark observation.

This is the filtering problem in miniature. The agent holds a belief: a probability distribution over the hidden state of the world. Each time step, two things happen. The world evolves according to dynamics the agent does not fully control or observe, so the belief must be pushed forward through the transition model. Process noise often spreads that predictive belief, although deterministic dynamics can also compress it. Then a new observation arrives, and the belief is corrected by weighting states according to how likely they would have produced that observation. Averaging over the observations the model itself predicts, conditioning reduces uncertainty, but a surprising individual observation can leave a realized posterior more uncertain than the preceding belief. The classical filters below implement this predict-then-correct alternation explicitly. The learned recurrent update later approximates the composed recursion in one forward computation and need not expose a separate predictive belief.

The phrase "all the agent needs" is doing real work. For a specified POMDP, a correctly maintained belief is sufficient for predicting future states and observations and for computing decision-relevant quantities such as the optimal value function. It does not preserve arbitrary facts about the past, such as the exact observation sequence or its length. To appreciate why this is useful, consider what the raw history looks like: a list of observations and actions that grows without bound as time passes, with no obvious way to compare two different-length histories. The belief replaces the decision-relevant information in that history with a single object of fixed type, a distribution over states. This is why the belief state matters beyond estimation. A POMDP can be rewritten as a belief-space MDP, an ordinary Markov decision process whose states are distributions, because the exact belief update below depends on the past only through the current belief and the new action-observation pair. Filtering is therefore not a preprocessing step bolted onto control; it is what restores the Markov property that planning and reinforcement learning require. Without it, value functions would need to condition on entire histories, and the dynamic programming machinery that makes sequential decision problems solvable no longer applies directly.