Offline and Conservative Model-Based RL › Pessimism and Conservative Objectives

The previous section diagnosed the disease: an unconstrained estimate is maximised by the optimiser, and off-support the estimate is arbitrary. This section describes the cure. The idea is almost embarrassingly simple, but its theoretical consequences are substantial, and the design choices around it are where most of the engineering lives.

Let $\mathcal{M}$ be a set of MDPs consistent with the data, in the sense that no MDP in the set is ruled out by the observed transitions. Offline, the data cannot distinguish them, so the honest thing to do is to consider the worst case within the set:

$$V^{\pi}_{\text{pess}}(s) = \min_{M \in \mathcal{M}} V^{\pi}_M(s).$$

This is pessimism. It says: if the true environment lies in the candidate set, I will plan for the candidate that treats me worst. Under that condition, the pessimistic value is a lower bound on the true value of a fixed candidate policy; how informative that bound is depends on the set. The corresponding minimax formulation has the agent choose a policy, "nature" choose an MDP from the specified set, and the agent receive the worst-case value. This is a planning objective, not a guarantee of risk-free outcomes.

In practice $\mathcal{M}$ is never enumerated. Instead, the worst case is approximated by subtracting a penalty from the reward:

$$\tilde r(s,a) = \hat r(s,a) - \lambda\, u(s,a),$$

where $u$ is the uncertainty function from the previous section and $\lambda$ is the pessimism coefficient. The penalty is called "conservative" because it reduces the appeal of uncertain regions relative to their unpenalised estimates. A high predicted reward can still outweigh a high penalty; this objective does not impose a confidence threshold. The intuition is that the penalty proxies the gap between $\hat r$ and the worst-case reward over $\mathcal{M}$, and where uncertainty is large that proxy can be large.

The penalty is not automatically a guarantee. To obtain a lower bound, it must cover the model's possible overestimation at every relevant state-action pair. Here is a sufficient condition for a fixed deterministic policy.

Suppose the true and learned Bellman operators differ by at most $\epsilon$ when applied to the true value function:

$$\big|(\mathcal{T}^\pi V_M^\pi)(s) - (\hat{\mathcal{T}}^\pi V_M^\pi)(s)\big| \le \epsilon \quad \text{for all } s.$$

The contraction property then gives $\|V^\pi_M-V^\pi_{\hat M}\|_\infty\leq\epsilon/(1-\gamma)$. More locally, let the penalised model Bellman operator be $\hat{\mathcal T}_{\rm pen}^\pi V(s)=\hat r(s,\pi(s))-\lambda u(s,\pi(s))+\gamma\mathbb E_{\hat P}V(s')$. Its value, averaged over the initial distribution, is

$$\hat J_{\text{pen}}(\pi) = \hat J(\pi) - \frac{1}{1-\gamma}\,\mathbb{E}_{(s,a)\sim d^\pi_{\hat M}}\big[\lambda\,u(s,a)\big].$$

For a pointwise lower bound, a sufficient condition is

$$\lambda u(s,\pi(s))\;\geq\; (\hat{\mathcal T}^\pi V_M^\pi)(s)-(\mathcal T^\pi V_M^\pi)(s)\quad\text{for every }s.$$

Under this condition, $\hat{\mathcal T}_{\rm pen}^\pi V_M^\pi\leq V_M^\pi$. Monotonicity and contraction of the discounted Bellman operator imply

$$\hat V^\pi_{\text{pen}}(s) \le V^\pi_M(s) \quad \text{for all } s.$$

The pessimistic estimate is a certified lower bound only when the stated condition holds. Since $V_M^\pi$ and the true Bellman error are unknown offline, a count or ensemble score is not automatically a valid bound on that error. Policy-improvement guarantees require additional coverage, estimation, and optimisation assumptions; the displayed inequality alone does not give a suboptimality rate.

Two things follow. First, a penalty that is too small to cover overestimation lacks this lower-bound guarantee, whereas one that is too large can suppress useful policies. Second, a uniform penalty calibrated to a global worst case taxes even well-understood steps. Here is the deep problem: the condition depends on $V^\pi_M$, which you do not have. A generic bound is $|V^\pi_M|\leq R_{\max}/(1-\gamma)$; tighter local bounds require additional assumptions. Using only the generic bound can produce the vacuous guarantee from the worked example earlier. More selective methods restrict where pessimism enters, and the rest of this section compares those choices.

MOPO. Model-based Offline Policy Optimization learns an ensemble of probabilistic dynamics models. Its practical uncertainty score takes the largest Frobenius norm of a member's predicted Gaussian scale matrix (described in the paper as maximum standard deviation), $u(s,a)=\max_i\|\Sigma_i(s,a)\|_F$; it subtracts $\lambda u$ from reward and trains a policy with MBPO-style short model rollouts. That score is heuristic rather than the admissible model-error upper bound required by the paper's theorem. The theoretical comparison also uses the true reward function in its penalized model; the practical implementation learns a reward model. Under the theorem's function-class, exact-reward, and admissible-dynamics-error assumptions, the result compares the learned policy with any candidate policy by

$$J_M(\hat\pi) \;\geq\; \sup_\pi\big\{J_M(\pi)-2\lambda\bar u(\pi)\big\},$$

where $\bar u(\pi)$ is the discounted sum of $u(s,a)$ under policy $\pi$ in the learned model. This is a conditional guarantee, not a promise that the practical Gaussian scale (standard-deviation) heuristic upper-bounds real model error.

The gap between the theorem and the algorithm is instructive. The theorem assumes an admissible bound on relevant dynamics error and an exact reward; the practical algorithm estimates a Gaussian scale score and a reward model from data. That heuristic may work empirically, but it does not inherit the theorem's guarantee without additional calibration and reward-error assumptions. Read the proof as a statement about what the penalty would need to control, not a certification of the fitted ensemble.

MOReL. The Model-based Offline Reinforcement Learning algorithm of Kidambi and colleagues takes a harder line. It partitions the state-action space into known and unknown using ensemble disagreement, and then constructs a pessimistic MDP in which unknown pairs lead to an absorbing terminal state carrying a penalty:

$$P_p(s_{\text{halt}}\mid s,a)=1\quad\text{for unknown }(s,a),\qquad r_p(s_{\text{halt}},a)=-\kappa.$$

The halt state is absorbing and carries the negative reward at every subsequent step; known pairs use the fitted transition. Thus its value under the displayed convention is $-\kappa/(1-\gamma)$, not a one-time $-\kappa$. This hard unknown-pair rule prevents a model rollout from inventing a favorable continuation after the halt, whereas a soft penalty allows it to continue. MOReL's algorithm optionally uses a behavior-cloned policy to initialize its planner; that is not a required behavior-cloning regularizer in its objective.

COMBO. Conservative Offline Model-Based Policy Optimization learns a Q-function from a mixture of real transitions and model rollouts. Its conservative term pushes down Q-values on state-action pairs sampled under model rollouts and up Q-values on logged state-action pairs. Thus it does not apply pessimism only to real data. Unlike MOPO's practical reward penalty, this regularizer acts on the value function and does not require an explicit model-uncertainty estimate.

The COMBO idea is subtly different from the others. It does not need to know where the model is uncertain; it can simply compare where the Q-function is being trained on real data versus rollout data, and it penalises the difference. This makes COMBO less dependent on the quality of an uncertainty estimator, but more dependent on having a good critic architecture and optimiser.

RAMBO. Instead of subtracting a hand-designed uncertainty score from reward, RAMBO (Rigter and colleagues) trains the model adversarially against the value function while also fitting the data. Its model objective still has a coefficient $\lambda$ controlling the adversarial term, so pessimism has not become hyperparameter-free. It is a value-aware model-learning approach rather than an uncertainty-penalty approach. The trade-off is that the model is trained both for data fit and for appropriately pessimistic values, which adds training complexity.

It is worth knowing that the same ideas appear on the model-free side, often with different vocabulary.

• BCQ (Batch-Constrained Q-learning) trains a generative model $G_\omega(s)$ over the dataset's actions and restricts the policy to perturbing samples from it. The constraint is on the support of the action distribution.
• BEAR applies an MMD constraint between the learned policy's action distribution and the behaviour policy's during actor policy improvement; it does not add MMD to the critic's objective.
• TD3+BC combines Q-maximisation with a plain action-regression term in the actor objective: $\max_\pi\,\mathbb E_{(s,a)\sim\mathcal D}[\lambda Q(s,\pi(s))-\|\pi(s)-a\|^2]$, with $\lambda=\alpha/\mathbb E_{(s,a)\sim\mathcal D}|Q(s,a)|$. Thus $\alpha$ scales the Q term through $\lambda$, not the regression term. It is startlingly simple and competitive.
• AWAC and AWR perform advantage-weighted behaviour cloning: the policy update is a weighted maximum-likelihood fit where the weight is $\exp(A(s,a)/\beta)$. Actions with low advantage get near-zero weight, so the actor update moves toward better logged actions. That statement does not guarantee that every critic backup, including AWAC's, avoids querying actions outside the dataset.
• IQL removes the OOD action query entirely by estimating the value function with expectile regression back-ups, which never need $\max_a Q(s,a)$. It is arguably the cleanest statement of the underlying principle: do not ask the model a question the data cannot answer.

All of these address the same distribution-shift problem from different sides: pessimism discounts uncertain estimates, while a policy constraint limits which estimates are queried. They are complementary design choices, not generally a formal dual pair. The model-free family does not require a dynamics model, yet it still faces coverage problems when a critic is asked to value unsupported actions.

The implementation is one line. Everything visible in the previous section came from adding - PENALTY * u_hat to the reward.

The choice of $\lambda$ is the interesting part. Below some threshold, the penalty is too weak to overcome the model's optimism and the planner still walks off the data. Above the sampled threshold in this toy grid, the greedy policy stays on the corridor and reaches the goal; this is not a general safety guarantee. Push it much higher and you do not get a better policy, only a more pessimistic value estimate, which matters when you use that estimate to choose between candidate policies rather than to act. This is why tuning $\lambda$ is subtler than it first appears: the optimal $\lambda$ for shaping a policy may be different from the optimal $\lambda$ for evaluating one.

To see the threshold, sweep the positive values of $\lambda$ across about 3.3 orders of magnitude (with zero as an additional baseline), solve the penalised model at each value, extract the greedy policy, and evaluate that policy exactly in the true grid.

Over-conservatism deserves more than a sentence, because it is the failure mode practitioners hit most often and it does not look like failure.

In the clean gridworld above, over-conservatism is invisible in the realised return. The corridor is the only route that pays anything, so increasing the penalty leaves nonterminal behavior and realised return unchanged in this example. In a realistic environment the pattern can differ. A dataset might cover a mediocre but safe region densely and a rewarding but thinly covered region sparsely. As $\lambda$ grows, the latter pays a larger penalty even if the model's uncertainty score is bounded. The agent can retreat to the safe region and produce a stable-looking policy that misses attainable reward.

The mechanism is a tax that rises as logged coverage falls, regardless of the region's actual reward. Well-sampled regions pay almost nothing; thinly sampled regions pay much more. This biases the policy toward the well-sampled region even when the well-sampled region is not where the reward lives. The result is not a broken policy. It is a policy that looks reasonable, that runs cleanly, and that quietly underperforms.

There are three symptoms worth watching for:

• Return collapse relative to behaviour cloning. A policy that fails to beat a behaviour-cloning baseline may be over-constrained or over-penalised, though weak coverage, model error, or optimisation failure can produce the same symptom. The baseline is a useful diagnostic, not proof of the cause.
• Sensitivity of the chosen policy to $\lambda$ across the middle of the range. If the argmax policy changes shape when $\lambda$ moves from 0.3 to 0.5, the penalty is not just taxing the estimate, it is deciding the policy. That is fragile.
• A large gap between penalised and unpenalised model scores. In this fixed model, the optimal penalised score cannot increase as $\lambda$ rises. A negative penalised score shows that the objective is heavily taxed; it does not imply the task's true return is negative or that the model believes the task is worthless.

The remedy depends on the cause. Reducing $\lambda$ may recover a good policy but can also reintroduce model exploitation. Better data about the promising region, where collection is allowed, can reduce uncertainty; longer training on the same unsupported data cannot manufacture that missing evidence. A better-calibrated local penalty or constraint can distinguish well-supported and thinly supported regions. MOPO's penalty varies with $(s,a)$; MOReL halts on pairs classified as unknown. Neither automatically calibrates uncertainty to true model error.

An optimiser can seek out overestimated actions, so upper-tail value errors are especially dangerous. Underestimation also matters: it can discard a good policy or prevent improvement beyond the behaviour policy. Conservative methods trade some of that opportunity for protection against unsupported optimistic choices. Whether the trade is appropriate depends on coverage and the decision at hand.