Offline and Conservative Model-Based RL › Offline and Conservative Model-Based RL

Picture a warehouse robot that has been running for three years. It has driven several thousand kilometres, followed human-designed routes, bumped into the same four pallets, and logged every state, action, and outcome. You now have a large dataset. You have no way to collect more. Can those logs support a better controller than the one that produced them?

The answer is not obvious and not always yes. The dataset was produced by some behaviour policy, perhaps a careful human operator or an older controller. Every transition reflects choices that policy made. If it never tried to squeeze between two pallets, the dataset contains no direct evidence about what happens when you try. If it never braked hard on a wet floor, the dataset is silent on that too. Improving the controller may require different actions, and those actions may lack coverage in the logs. That is the tension we spend the chapter unpicking.

Every model-based method we have covered so far in Part VIII: Decision-Centric Research Lineages assumed a closed loop. PETS, MBPO, Dreamer, MuZero, and TD-MPC all interleave acting in the world with learning from the consequences. The world model is wrong in a hundred small ways, the agent tries something, reality corrects it, and the model improves. That correction signal is the engine.

When the agent acts, the observed successor provides a targeted new sample at the state-action pair it tried. In a stochastic environment, one outcome is not a measurement of how wrong the model's predictive distribution is; repeated observations help distinguish environmental randomness from systematic model error. Over many interactions a well-designed learning loop can concentrate model corrections where the agent visits. The state distribution induced by the policy and the region where the model improves can then move together. Offline learning loses that feedback loop.

Offline reinforcement learning removes that engine during training. The agent gets one fixed dataset and no new interaction while learning its policy. Deployment may provide feedback later, or an offline-to-online phase may collect more data, but neither can correct an unsupported decision before this offline training stage ends. The policy must therefore be selected using evidence already in the log.

That single change breaks almost everything. Two failure modes follow directly from it, and they are worth stating plainly before any mathematics:

• The log alone does not constrain the model off its support. If your dataset never contains the state-action pair $(s, a)$, competing predictions there can fit the logged transitions equally well. A model that says "you teleport to the goal" and a model that says "you fall into a pit" may both fit the training set. Architecture, prior knowledge, or justified structural assumptions can distinguish them, but the log provides no direct target at that pair to pull the model toward the truth.
• The optimiser can exploit unsupported optimistic estimates. A policy improvement step that maximises estimated value favours regions with high estimates. If the model happens to be optimistic off-support, that step can select a policy that visits a region where the log provides little evidence. This is model exploitation, and it is not a defect of one algorithm: it is a risk whenever policy selection relies on estimates that are insufficiently grounded.

The mitigations address those two risks. We can quantify how uncertain the model is (ensembles, counts, densities). We can be pessimistic, subtracting an uncertainty penalty to make poorly supported choices less attractive. Or we can constrain the policy itself to stay near the behaviour that produced the data. The rest of the chapter develops all three ideas, illustrates uncertainty and pessimism on a gridworld, and then turns to the meta-problem that offline RL never escapes: how do you evaluate a policy you cannot run?