Offline and Conservative Model-Based RL › Policy Constraints and Offline Evaluation

Pessimism and constraints are two responses to distribution shift, often used together rather than as formal duals. This section covers the constraint family, then turns to a separate problem: how can you judge a learned policy without running it?

Pessimism reduces estimated values where a penalty signals model risk, seeking to discourage unsupported choices. A support constraint instead restricts the policies considered. Neither automatically ensures that the resulting policy stays near the data: a heuristic penalty can miss model error, and an approximate constraint can admit poorly covered actions. They act on different parts of the optimization problem, and their effectiveness depends on their assumptions and implementation.

A support constraint restricts $\pi$ to the set of policies whose state-action distribution is covered by the dataset:

$$\pi \in \Pi_{\text{supp}} = \Big\{ \pi : d^\pi(s,a) > 0 \Rightarrow \rho_\beta(s,a) > 0 \Big\}.$$

This is an idealized constraint on the full state-action occupancy, including states reached by the target policy. BCQ's conditional generative model and bounded action perturbation approximate the behavior-action support at queried states; they do not enforce this global occupancy condition, especially at unseen states. Under a continuous-density behaviour distribution, each individual sampled point has probability zero even when the distribution has full-dimensional support. That conditional support is hard to infer from a finite log, so practical methods use divergences or generative action models as approximations.

A behaviour constraint is softer: rather than requiring full support, it penalises divergence from the behaviour policy:

$$\max_\pi \; \mathbb{E}_{s\sim\mathcal{D}}\Big[\mathbb{E}_{a\sim\pi(\cdot\mid s)}Q(s,a) \;-\; \beta\,D\big(\pi(\cdot\mid s)\,\|\,\pi_\beta(\cdot\mid s)\big)\Big],$$

where $D$ denotes a discrepancy or penalty, such as KL, MMD, or squared action distance. BEAR constrains its actor with MMD, while TD3+BC uses an action-regression term. AWAC instead performs advantage-weighted behaviour cloning; its update is not literally this displayed divergence objective. The choice matters: KL is asymmetric, MMD compares distributions through a kernel, and squared distance ignores much of the action distribution's structure.

These mechanisms can be complementary, but no general equivalence follows. A support constraint may still allow optimistic errors within its permitted set; a pessimistic critic may still be poorly calibrated where data are scarce. CQL applies conservatism to critic values, MOPO penalises rewards in model rollouts, and MOReL changes unknown transitions to lead to HALT. BEAR and TD3+BC illustrate direct actor restrictions. Specific implementations may combine mechanisms, but the papers do not establish a universal constraint–penalty duality.

Here is the part of offline RL that practitioners consistently underestimate. You have produced a policy. You cannot run it. You must decide whether to deploy it, and if you have several candidates, which one to pick.

The candidates are usually not just policies. They are (policy, model architecture, ensemble size, rollout length, penalty coefficient, learning rate) tuples, and the number of combinations can be large. Selecting among them is offline model selection, a separate problem that can be difficult with only logged data. A model-based estimate of a policy's value can inherit errors from the model used for learning. If that model overestimates a region, it may also overestimate policies that visit the region and favor them in selection. This is a related form of model exploitation on the evaluation side.

Three families of estimators exist.

Importance sampling and its variants. Given $m$ independent trajectories from a known behaviour policy $\pi_\beta$, reweight each discounted trajectory return $G_i=\sum_{t=0}^{T_i-1}\gamma^t r_{i,t}$ by the ratio of action probabilities:

$$\hat J_{IS}(\pi) = \frac{1}{m}\sum_{i=1}^{m} \left(\prod_{t=0}^{T_i-1} \frac{\pi(a_{i,t}\mid s_{i,t})}{\pi_\beta(a_{i,t}\mid s_{i,t})}\right) G_i .$$

Under matching initial-state and environment distributions, known behaviour propensities, target-policy support within behaviour support, and finite expectation, ordinary trajectory IS is unbiased for the return represented by each complete sampled trajectory. For the infinite-horizon $J(\pi)$ defined above, the recorded trajectory must include all nonzero rewards (for example, by ending in a zero-reward absorbing state) or use a valid tail correction; otherwise this formula estimates a truncated return. Its variance can grow exponentially with horizon when the policies differ enough, but this is not inevitable. Per-decision IS (PDIS) applies prefix ratios to individual rewards and is unbiased for the same return target under the corresponding support, sampling, and trajectory-completeness conditions; weighted/self-normalised IS trades finite-sample bias for possible variance reduction. Doubly robust estimators combine an approximate value function with importance weights. Long horizons and continuous actions can make ratio-based estimators impractical when overlap is weak, but their suitability depends on the policies and data rather than horizon alone.

Fitted Q-evaluation. Learn $Q^\pi$ by iterating fitted Bellman back-ups on the offline dataset, using actions from $\pi$ at the successor state rather than an argmax. The resulting value estimate depends on approximation, optimisation, and evaluation-policy coverage; a policy produced by an offline algorithm is not by construction covered by the data. FQE avoids multiplying whole-trajectory probability ratios, but fitting a critic can be computationally more expensive than computing IS weights. It is a useful OPE baseline when its assumptions and implementation are stated, not an automatic certificate for every D4RL or RL Unplugged task.

Model-based evaluation. Evaluate a fixed candidate policy in the learned model. Holding the policy fixed removes an additional maximisation within that evaluation calculation, but it does not erase how the policy was selected. A policy optimised on the same model may already exploit its weak spots, and choosing the highest of many model-based estimates can do so again. Model-based evaluation is useful when model accuracy is supported along that policy's occupancy, with uncertainty and coverage reported alongside the estimate.

It also has an important failure mode, and it is the same one. The evaluator shares the model's blind spots. If the model has a large optimistic region and the candidate policy goes there, the evaluator reports a high return for exactly the policy you should not deploy. Techniques that help include averaging over the ensemble, reporting the worst-case estimate across members, and using the uncertainty map to flag evaluations whose trajectories enter high-$u$ regions.

The discounted unvisited-state occupancy diagnostic shows that 45.2% of the naive candidate's model-rollout mass lies in cells absent from the dataset, versus 0.0% for the pessimistic candidate. The unpenalised learned-model evaluation of the pessimistic policy is about 0.558, while its penalised planner score is about 0.509. The naive policy's model estimate relies partly on unobserved dynamics and should not be read as a guarantee about its real return.

This is the practical lesson for offline evaluation: pair estimates with coverage diagnostics and sensitivity checks. This state-only fraction flags the naive policy's exposure to unvisited cells, but misses unsupported actions within visited states. Inspect discounted state-action occupancy or logged pair counts as well; neither check by itself certifies model accuracy or the value estimate.

The standard offline RL benchmarks are:

• D4RL. The benchmark includes MuJoCo locomotion, AntMaze, Franka kitchen, and Adroit. Locomotion datasets use labels such as random, medium, medium-replay, medium-expert, and expert; those labels do not apply uniformly to the other task families. Medium-quality locomotion data provide a useful improvement-from-imperfect-behaviour test, but no one quality label is universally the hardest across tasks and methods.
• RL Unplugged. A suite spanning control and games, including continuous-control and Atari-style settings, with an emphasis on realistic logged data collection.
• NeoRL. Datasets collected by policies of controlled quality, designed for studying near-realistic offline settings where the behaviour policy is neither random nor expert.

Two protocol points matter more than the choice of benchmark. First, report the benchmark's prescribed score normalization where it is defined; D4RL commonly normalizes scores against reference random and expert returns so raw returns across tasks are more comparable. Second, report the variation across seeds and the aggregation statistic. Mean and median can differ substantially, so a ranking that changes with aggregation deserves scrutiny.

For rollout-based methods such as MOPO and COMBO, one common protocol is to fix the dataset, fit a model per seed, generate short model rollouts, train a policy or critic using real and imagined data, and choose the final policy with an offline criterion. MOReL instead plans in its pessimistic learned MDP, and Trajectory Transformer plans over predicted sequences; the rollout-mixture recipe does not describe every model-based method. In rollout methods, horizon is a consequential hyperparameter: a short rollout limits direct propagation of model error but leaves more work to the critic, while a long rollout can accumulate unsupported predictions. Policy selection needs its own reported criterion because no online return is available to settle competing settings.