Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Mathematical and Dynamical Foundations › Worked Example: A Thermostat with Memory

Michael BrenndoerferJune 6, 20266 min read

World Models HandbookAll chapters

Part 1 of 12The World-Model Idea

Chapter 6Mathematical and Dynamical Foundations › Worked Example: A Thermostat with Memory

Reading section 6 of 11 · Worked Example: A Thermostat with Memory · 6 minSection 6 of 11 · 6 min

Section 6 of 11 · 6 min read

Mathematical and Dynamical Foundations › Worked Example: A Thermostat with MemoryOne section · Sections in this chapter6 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

It is time to make the running example concrete. The system is a room with a heater, governed by a discrete-time transition:

T_{t+1} = T_t + \alpha\, m_t - \kappa\,(T_t - T_{\text{out}}) + \varepsilon_t,

where $T_t$ is room temperature, $m_t \in \{0, 1\}$ is the heater command, $\alpha = 0.6$ is the heating rate, $\kappa = 0.02$ is the cooling coefficient, $T_{\text{out}} = 8$ degrees is the outside temperature, and $\varepsilon_t$ is small thermal noise. The physics is deliberately minimal: linear heat flow with a constant outside sink, a constant heater input, and additive noise. Nothing here is hard to simulate, which is exactly why any modeling failure we observe later will be attributable to representation and dynamics rather than to simulation complexity.

The controller uses a hysteresis rule: turn the heater on when the temperature falls below 19.5 degrees, turn it off above 20.5 degrees, and otherwise keep the previous command.

Hysteresis is the interesting ingredient. Without it, the controller would flicker on and off at the threshold. With it, the controller's decision at the intermediate temperatures depends not only on the current temperature but on which side the system approached from, that is, on the past. Watch what this does to the dynamics.

In[19]:

Code

ALPHA = 0.6
KAPPA = 0.02
T_OUT = 8.0
NOISE_SD = 0.05
LOW, HIGH = 19.5, 20.5


def controller(temp, mode, low=LOW, high=HIGH):
    """Hysteresis control: 1 = heat, 0 = off. Keeps previous command in the deadband."""
    if temp < low:
        return 1
    if temp > high:
        return 0
    return int(mode)


n_steps = 2000
temps, modes = [], []
temp, mode = 20.0, 0

for t in range(n_steps):
    cmd = controller(temp, mode)
    temp = (
        temp + ALPHA * cmd - KAPPA * (temp - T_OUT) + rng.normal(0.0, NOISE_SD)
    )
    mode = cmd
    temps.append(temp)
    modes.append(mode)

temps = np.array(temps)
modes = np.array(modes, dtype=float)

n_switches = int(np.sum(np.diff(modes) != 0))
mean_period = 2.0 * n_steps / max(n_switches, 1)

Out[20]:

Console

Heater duty cycle (fraction on): 0.402
Number of on/off switches:       450
Mean cycle period:               8.9 steps
Temperature range: [19.16, 20.96] degrees

The system does not settle to a fixed point. Instead it locks into a repeating oscillation between the two thresholds: a limit cycle attractor created by the interaction between the linear thermal dynamics and the nonlinear switching controller. Neither component alone would oscillate: the thermal dynamics alone relax exponentially to a steady temperature, and the switching logic alone is just a lookup table. The oscillation lives in their coupling, which is typical of limit cycles in engineered systems. The measured period and duty cycle match what the physics suggests: heating near 20 degrees proceeds at roughly $0.6 - 0.02 \times 12 \approx 0.36$ degrees per step, cooling at roughly $0.25$ degrees per step, so each half-cycle takes a handful of steps. The asymmetry between those rates is why the sawtooth below leans: fast climbs, slower descents.

Out[21]:

Visualization

Long temperature time series oscillating between two threshold lines. — Room temperature under hysteresis control settles into a sustained limit cycle between the two thresholds.

Zoomed sawtooth oscillation crossing the two thresholds. — A zoomed window shows the asymmetric sawtooth: slower cooling segments alternate with faster heating segments.

Now the punchline about state. Suppose you try to predict $T_{t+1}$ from $T_t$ alone. During the deadband between the thresholds, the same temperature can be approached with the heater on or off, and the two cases lead to next-step temperatures differing by the full heating increment $\alpha = 0.6$ degrees. Temperature alone is not a sufficient state: the pair $(T_t, m_t)$ is, because given both, the next temperature is fully determined up to noise. The hidden variable here is the heater mode, a one-bit memory that the hysteresis rule makes unpredictable from temperature alone. Note how economical the fix is: infinitely much history collapses into a single bit, because the hysteresis rule ensures the mode is fully determined by the recent past and fully determines the relevant future. Not every hidden state is so cooperative, but the pattern, find the small object that summarizes the past, is universal.

We can see the insufficiency directly in the data. Collect all logged steps whose current temperature falls in a narrow band inside the deadband, and split them by heater mode. Conditioned on temperature alone, the next temperature is bimodal, with the two modes separated by roughly the heating increment; conditioned on temperature and mode, each half collapses to a tight unimodal cloud sitting at the noise floor. This is exactly what a missing hidden state looks like in a conditional distribution.

In[22]:

Code

current_temps = temps[:-1]
next_temps = temps[1:]
transition_modes = modes[1:]
deadband = (current_temps >= LOW + 0.1) & (current_temps <= HIGH - 0.1)
on_mask = deadband & (transition_modes == 1.0)
off_mask = deadband & (transition_modes == 0.0)

hist_edges = np.linspace(19.4, 21.4, 45)
hist_centers = 0.5 * (hist_edges[:-1] + hist_edges[1:])
hist_width = hist_edges[1] - hist_edges[0]

dens_mixed = np.histogram(next_temps[deadband], bins=hist_edges, density=True)[
    0
]
dens_on = np.histogram(next_temps[on_mask], bins=hist_edges, density=True)[0]
dens_off = np.histogram(next_temps[off_mask], bins=hist_edges, density=True)[0]

Out[23]:

Visualization

Histogram with two peaks about 0.6 degrees apart. — Next temperature given only the current temperature inside the deadband: the distribution is bimodal because the heater mode is unknown.

Two overlaid narrow histograms, one per heater mode. — Splitting the same transitions by the heater command applied next yields two tight unimodal distributions separated by the heating increment.

This is the smallest possible instance of a problem that pervades the field. Partially observable systems hide state variables inside their histories, and agents that ignore the hidden component see a world that appears to violate the Markov property even when the underlying world is perfectly Markovian. The remedies, carrying beliefs over hidden states or learning memory-based representations, are major subjects of Part II and Part III. Here, we can simply measure the damage.