Active Perception, Dual Control, and Exploration › Limitations and Impact

The value-of-information gap is a practical liability. Raw information gain and prediction-error curiosity can be misaligned with task value. An agent may resolve an unreachable target's identity or an irrelevant visual detail while missing uncertainty that changes the action. Belief-MDP planning and IDS-style information ratios make the decision link explicit, but they are usually more expensive. Practical systems therefore treat intrinsic reward as a surrogate and evaluate it on the extrinsic task.

Persistent prediction error and noisy TV are related proxy failures with different mechanisms. Noisy TV is driven by aleatoric unpredictability. Persistent RND or model error can also come from limited capacity, optimization, nonstationarity, or nuisance features. Remedies therefore differ: inverse-dynamics features can suppress uncontrollable pixels, explicit aleatoric modeling can separate noise from model uncertainty, and learning-progress rewards can pay only when error decreases. Ensemble disagreement is only an epistemic proxy and must be calibrated rather than assumed correct.

Bias from passive data. As covered in [Interaction Data and Passive Observation], intrinsic rewards depend on the data distribution, and an agent that only ever collects the data its current policy induces has a self-reinforcing sampling bias. It will fail to discover regions it never visits and cannot assess the value of information it never obtains. Off-policy and exploratory priors help, but they do not eliminate the issue. A model that is overconfident in unvisited regions also makes probing look uninformative, which further suppresses exploration. This is the exploration collapse failure mode: the agent becomes certain it has nothing left to learn while being wrong.

Safety guarantees are conditional on their assumptions. SafeOpt, CBFs, shields, chance constraints, and recovery policies use different assumptions: safe seeds, regularity, sound abstractions, calibrated confidence sets, feasible controls, disturbance models, or certified backups. Model-based filters inherit model mismatch, especially near poorly observed boundaries. In a high-stakes setting, state the exact guarantee and probability space, preserve a validated fallback where available, and do not treat a learned score as a certificate.

Combinatorial action spaces and long-horizon exploration. One-step information gain is easy to state but useful observations may require several coordinated actions. Belief-space planning represents that horizon but faces branching and dimensionality; intrinsic bonuses compose with scalable policy learners but solve neither dimensionality nor long-horizon credit assignment by themselves. Hierarchical methods in [Hierarchical, Symbolic, Language, and Multi-Agent Planning] provide another way to structure the search.

Evaluation is subtle. Intrinsic reward changes the policy and the data distribution, so its training value is not task performance. Evaluate extrinsic return, safety events, uncertainty calibration, and held-out tasks against an exploration-free baseline. Decision-relevant state-action coverage can be a useful diagnostic, but it does not guarantee generalization. Seed sensitivity, tuning budgets, baselines, normalization, distribution shift, and statistical power can all change a comparison.

Active perception and dual control offer a common formal lens for closed-loop sensing, system identification, viewpoint selection, sparse-reward exploration, and experiment design. Curiosity has enabled important exploration results, while many model-based RL systems instead rely on random collection, MPC, ensembles, or short model rollouts without an intrinsic bonus. Reward maximization and knowledge acquisition are different objectives; making them compatible requires an explicit decision model, scale, and safety criterion.