Failure Modes and Model Exploitation › Limitations and Impact

These risks can arise across architectures and training regimes; none is inevitable for every finite-data agent. Exact representable dynamics, sufficient information, an aligned objective, and controlled execution can eliminate particular mechanisms on a specified domain. The examples identify what to check, not an architecture-independent theorem that every agent fails.

Some defects admit direct repairs within a known scope. Align a missing task preference or enforce a justified hard safety constraint; a soft cost term alone does not guarantee safety. Expose sufficient decision information and ensure it is encoded and used; revealing one variable need not remove all aliasing. Faithful distribution modeling or correctly enforced support constraints can address the corridor error, but a generic probabilistic output head need not. Each repair still needs a test or argument for its stated scope.

Residual one-step errors can propagate, but compounding is not inevitable. If $\hat f=f$ and the initial states and actions match, discrepancy is zero at every horizon, even for $f(s)=2s$. For fixed maps on a fixed domain, $\varepsilon$ and $L$ constrain an envelope; refitting, structural constraints, changed operating regions, or feedback can change the comparison. Short rollouts and suitable self-correction can mitigate some residual errors. Calibrated uncertainty can inform decisions, not automatically repair the mean dynamics. Choose horizons using sensitivity, task requirements, and measured error rather than assuming every long-horizon model must eventually fail.

Exploitation is a joint interaction of scoring error, selection, and task consequences; this does not mean the model has no defect. Aggressive search, limited joint state-action coverage, and stochastic scores can create risks, but no universal worst-case ordering follows from those labels. Support restrictions, shorter horizons, and uncertainty penalties can involve tradeoffs. A penalty requires justified error coverage and a suitable coefficient for a conservative guarantee; arbitrary disagreement or variance is not automatically an error bound. Tradeoffs are not inevitable: implementing the actuator's known clipping removes this toy's scoring defect without removing any physically achievable transition. Ask which tested defects can be repaired, which residual risks remain, and what each proposed mitigation costs.

Prediction measurements alone need not establish reliable decisions. They must be connected to the intended task, available information, selected inputs, and execution protocol. Empirical checks are valuable when those properties are uncertain; fully specified finite or formal models may establish scoped guarantees by other means. The chapter does not quantify where most real deployment effort is spent.

The next chapter, Part XII Chapter 2 on robustness, calibration, and safe control, is planned to develop uncertainty-aware and constrained control responses. Here we have suggested directions, not validated a general mitigation package. The question is how to act under specified model and information limitations. Robust control, Bayesian decision theory, and formal verification provide relevant approaches; not every deployment requires every approach or has inevitable compounding error.