Robustness, Calibration, and Safe Control › Limitations and Impact

The techniques in this chapter are useful, but each has limitations, and a clear account of those limitations matters more than promotion. You may be worse off if you adopt these tools without understanding their limits, because the tools can create false confidence.

Stress testing depends on the mechanisms and ranges considered. Unanticipated couplings, sensor degradation, and rare conditions can remain outside the suite. Revisit it when incidents, near-misses, operating conditions, or requirements reveal missing cases. A calendar year without growth is not by itself evidence that the suite is inadequate; relevance and coverage matter more than its size.

Calibration is a property of specified forecasts and a population. Deployment may sample a population, but adaptive closed-loop data can be dependent and differ from the calibration population. Marginal coverage can hide subgroup failures, including safety-relevant states that need not be the least densely trained region. Broad aggregate diagrams provide some information, not a substitute for conditional and trajectory-level evaluation; see Angelopoulos and Bates.

Ensemble disagreement is not a general-purpose dynamics-shift detector. The tested input shift increased disagreement more than the damping shift, while aggregate one-step error changed little. This is evidence about one fitted ensemble and two populations, not a universal detector comparison. Resampling old data cannot directly observe a newly changed transition law, although changed trajectories may later create unfamiliar inputs that increase disagreement. Different ensemble constructions or shift sizes can behave differently; test the signal rather than treating it as a certificate.

Risk-sensitive planning depends on its rollout distribution. CVaR over a mis-specified model can optimize the wrong tail. With $M=9$ fits and two draws per fit, $S=18$ samples represent each candidate; at $\alpha=0.75$ the empirical upper-tail mass is $4.5$ samples. Four contribute fully and the fifth half. These samples share fitted members and are not eighteen independent estimates of model uncertainty. Increase draws and evaluate across independent training/evaluation seeds to assess sensitivity. Moderate risk levels reduce tail-estimation demands but do not establish safety.

Runtime assurance makes proof obligations explicit. Empirically tuned fallback behavior and convenient parameter ranges do not constitute a safety proof. An independent safety argument can avoid verifying the learned performance model, but still needs justified plant and disturbance assumptions, initial admissibility, sound decision logic, fallback continuation, and timing. That burden may be difficult rather than small or local. Simplex research illustrates aircraft and robot applications; earlier work also discusses pacemaker models. These are research examples, not a claim of universal certified deployment.

Model errors can affect both performance and safety; their consequences depend on the plant and control architecture. The experiments here distinguish empirical prediction diagnostics, tail-sensitive objectives, and fallback switching. They do not show that consequences are bounded in every deployment. Moving from this prototype to a safety argument requires explicit uncertainty/disturbance bounds or probability levels, validation of the fallback and decision module, and timing and state-estimation analysis. The value of the separation is that these obligations can be stated independently of a leaderboard prediction score.

In Part XI: Evaluation and Understanding, prediction was an evaluation axis. This chapter adds closed-loop consequences under stated stresses. These rankings need not agree. Deployment readiness depends on actual requirements and validated operating conditions, not on a prediction score alone or a universal ranking of survival.