Active Perception, Dual Control, and Exploration › Exploration with Safety Constraints

Exploration is the deliberate acquisition of information, and information acquisition can hurt. A house-cleaning robot that explores a staircase learns about the stairs; it may also fall. An autonomous car that explores aggressive steering learns about the tire model; it may also leave the road. Safety-constrained exploration asks how an agent can learn the cheapest possible lessons about the hardest constraints. The theoretical guarantees in this area are thinner than elsewhere in the chapter, so we distinguish carefully among approaches and their assumptions.

A standard formalism is the constrained MDP (CMDP). It adds a set of cost functions $C_i(s, a)$, each with a budget $c_i$:

$$\max_\pi \; \mathbb{E}_\pi\!\left[ \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \right] \quad \text{subject to} \quad \mathbb{E}_\pi\!\left[ \sum_{t=0}^{\infty} \gamma^t C_i(s_t, a_t) \right] \le c_i \;\; \forall i,$$

with $\gamma\in[0,1)$ for bounded infinite-horizon returns.

Costs encode quantities to constrain, such as collision indicators, energy, torque, or risk exposure. An expected discounted-cost budget can permit rare costly trajectories while satisfying the expectation. That criterion is distinct from cumulative violation regret, a chance constraint, an almost-sure constraint, or robust pathwise invariance. None implies another without a specific theorem and assumptions, so a safety claim must name both the event being bounded and the probability space.

A hard constraint is intended to hold at every step rather than only on average. A controlled-invariant safe set $\mathcal{S}_{\text{safe}}$ is one for which there exists an admissible feedback policy that keeps the trajectory inside for all disturbances covered by the model. Requiring the set to remain safe under every admissible control is the stronger notion of strong invariance; robustness instead refers to preserving the relevant invariance property under the specified disturbances or model uncertainty. Two canonical tools are:

• Control barrier functions (CBFs): a scalar certificate $h$ whose superlevel set $\{s:h(s)\ge 0\}$ defines the candidate safe set. Under regularity assumptions, if a feasible control satisfies a condition such as $\dot h(s,a)\ge-\alpha(h(s))$ for an appropriate class-$\mathcal K$ function $\alpha$, that set can be forward invariant for the modeled dynamics. A CBF need not be a signed distance. With state uncertainty, one conservative option enforces the condition over a confidence set; chance-constrained and belief-space barrier designs encode uncertainty differently.
• Shielding / safety filters: take the exploratory action $a_{\text{exp}}$ and test its feasibility (against a model, a reachability computation, or a hard-set controller). A filter may accept, repair, or replace the proposal with an admissible action; projection is one possible repair, not the definition. This interlock permits exploration within its modeled constraints when a feasible safe response exists. For model-based agents the filter is itself model-based, so its guarantee is only as good as the model (a model mismatch failure mode we discuss in the reliability and safety chapters).

The safe exploration literature distinguishes several non-equivalent guarantee types:

• Expected-budget safety constrains expected cumulative cost and may still permit costly trajectories.
• Violation-regret guarantees bound cumulative excess or the number of violations under a particular online-learning theorem.
• High-probability or chance constraints bound a stated event with probability at least $1-\delta$. Depending on the theorem, the probability may cover estimation data, process and observation noise, policy randomness, or their joint law.
• Pathwise or invariant-set guarantees keep every modeled trajectory in a safe set under specified disturbances and a feasible controller.

SafeOpt belongs to the high-probability category: under its Gaussian-process regularity, confidence, and initial-safe-set assumptions, it restricts evaluations to points certified safe on the confidence event. It is not a sublinear-violations method.

These guarantees rest on method-specific assumptions, for example a safe seed set, regularity, calibrated confidence bounds, a sound abstraction, feasible safe controls, bounded disturbances, or a certified backup controller. A CBF-based controller is safe only if its barrier conditions hold for the true dynamics. Near a boundary, model mismatch can make the filter satisfy its modeled inequality while the physical system crosses the boundary. This is why guarantees in learned-model settings must state their assumptions next to the guarantee.

Safety and exploration can conflict in a structural way. Informative actions can lie near a known-safe boundary when that boundary also marks the edge of the data-supported region. This alignment is common in some safe-expansion problems but is not guaranteed. When it does occur, a small model error near the boundary can become a violation. Three ways practitioners resolve the tension are:

1. Explore from a known-safe baseline. A certified controller executes or recovers while a learned policy suggests probes. The permission rule must preserve enough recoverability margin; offline conservative value estimation is not itself a certified safety fallback.
2. Explore in simulation, deploy conservatively. This is common in some robotics applications, but simulator violations are not literally free: they still consume compute and can teach policies that exploit simulator errors. The main deployment risk is the sim-to-real gap.
3. Optimize a risk-sensitive surrogate. Soft penalties or a tail-risk objective such as CVaR can discourage hazardous probes. They change preferences but do not enforce a hard safety constraint.

Since information gain is computed under a probability model, and the consequences of probing are random, it is natural to make the exploration objective risk-sensitive. The standard device is the conditional value at risk (CVaR): for a random cost $Z$ and risk level $\alpha \in (0,1)$,

$$\mathrm{CVaR}_\alpha(Z) = \min_{\eta \in \mathbb{R}}\left\{\eta + \frac{1}{1-\alpha}\,\mathbb{E}\left[(Z-\eta)_+\right]\right\},$$

where $(x)_+=\max(x,0)$. For a continuous loss distribution with no atom at its $\alpha$-quantile, this equals the conditional mean above $\mathrm{VaR}_\alpha$; the optimization form remains valid for discrete distributions with mass at the quantile. A CVaR penalty or constraint controls average upper-tail cost. It can still choose a risky probe if the permitted bound or reward tradeoff favors it, so it does not guarantee zero tail exposure. Unlike variance, upper-tail CVaR targets large costs rather than deviations in both directions.

Classical optimism says "act as if the world is as good as it plausibly could be." For safety constraints, the analogue is the reverse: "act as if the world is as bad as it plausibly could be." These two principles routinely appear in the same algorithm, one governing reward and one governing cost. Mixing them up is a classic bug. Keep the sign conventions explicit in code: reward bonuses are added, cost bonuses are subtracted or bounded.

We now have all the conceptual ingredients: information gain as the knowledge objective, curiosity as the motivation to pursue it, and constraints as the limit on how far to go. The next section builds a small end-to-end system.