Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Mathematical and Dynamical Foundations › Information, Compression, and Sufficient Statistics

Michael BrenndoerferJune 6, 20268 min read

World Models HandbookAll chapters

Part 1 of 12The World-Model Idea

Chapter 6Mathematical and Dynamical Foundations › Information, Compression, and Sufficient Statistics

Reading section 5 of 11 · Information, Compression, and Sufficient Statistics · 8 minSection 5 of 11 · 8 min

Section 5 of 11 · 8 min read

Mathematical and Dynamical Foundations › Information, Compression, and Sufficient StatisticsOne section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Probability tells us how to represent belief; dynamics tells us how belief evolves. Information theory tells us what is worth remembering. For world models this is not an abstract luxury: the central engineering question of the field is which aspects of a potentially infinite history must be carried forward in a finite state, and information theory is the discipline that answers such questions. Every architecture choice in latent world modeling, from bottleneck widths to memory mechanisms, is implicitly a bet about which information is dispensable, and the tools in this section make those bets explicit and checkable.

Entropy: measuring uncertaintyLink Copied

The entropy of a discrete random variable measures its uncertainty in bits:

H(X) = -\sum_x p(x) \log_2 p(x).

A fair coin has $H = 1$ bit; a certain outcome has $H = 0$ ; a uniform draw from eight options has $H = 3$ bits. Entropy is also the answer to a compression question: $H(X)$ is the expected number of bits per symbol needed to encode draws from $X$ without loss, a result due to Shannon that anchors the deep connection between prediction and compression. Any system that predicts well can be turned into a compressor and vice versa, a duality that resurfaces in modern world models trained with likelihood objectives and in the coding-inspired analyses we will encounter in Part III. The intuition behind the coding theorem is worth holding onto: symbols you can predict need few bits, because the decoder can guess them, while surprising symbols need many. Prediction quality and code length are two views of the same quantity.

Conditioning reduces uncertainty on average. The conditional entropy $H(X \mid Y)$ measures the uncertainty remaining in $X$ after observing $Y$ , and it satisfies $H(X \mid Y) \le H(X)$ , with equality exactly when $X$ and $Y$ are independent. Observing something can never hurt, on average, though it can fail to help if it carries no relevant information. One technical caution: for continuous variables the direct analogue, differential entropy, can be negative and is not directly a bit count. In continuous settings, work with relative quantities such as mutual information, defined next, which behave cleanly.

Mutual information and the data processing inequalityLink Copied

Mutual information measures how much knowing one variable reduces uncertainty about another:

I(X; Y) = H(X) - H(X \mid Y) = \sum_{x, y} p(x, y) \log_2 \frac{p(x, y)}{p(x)\,p(y)}.

It is symmetric, nonnegative, and zero exactly when $X$ and $Y$ are independent. Unlike correlation, it detects nonlinear dependence: if $Y = X^2$ , correlation is zero but mutual information is large. This matters for world models because the dependencies that carry predictive signal, resonances, thresholds, switching regimes, are typically nonlinear, and a team that audits its representations with correlations alone will certify as "uninformative" exactly the features doing the heavy lifting.

The single most consequential fact for our purposes is the data processing inequality: if $X$ , $Y$ , and $Z$ form a chain in which $Z$ is produced from $Y$ without further access to $X$ , then

I(X; Z) \le I(X; Y).

No transformation of the data, however clever, can increase the information it carries about the source. Deterministic processing can only preserve or destroy information. Stochastic processing destroys information in expectation as well, since randomness injected downstream cannot retroactively clarify the upstream signal. This innocent-looking inequality governs what encoders, bottlenecks, and abstractions are allowed to do, and it is the reason "compression" and "information loss" are precise engineering terms rather than metaphors. Whenever an encoder maps a rich observation history into a compact latent, the inequality certifies that whatever the latent fails to capture about the source is gone for good, no matter how deep the networks stacked on top of it.

Let's verify the inequality empirically on a simple communication channel: a source $X$ drawn uniformly from eight symbols passes through unchanged to $Y$ , then degrades into $Z$ by additive noise.

In[16]:

Code

def mutual_information(counts):
    """Mutual information in bits from a contingency table of counts."""
    pxy = counts / counts.sum()
    px = pxy.sum(axis=1, keepdims=True)
    py = pxy.sum(axis=0, keepdims=True)
    mask = pxy > 0
    return float(np.sum(pxy[mask] * np.log2(pxy[mask] / (px * py)[mask])))


def count_pairs(x_vals, z_vals, x_card, z_card):
    counts = np.zeros((x_card, z_card))
    np.add.at(counts.ravel(), x_vals * z_card + z_vals, 1)
    return counts


n_channel = 200_000
source_entropy_bits = float(np.log2(8))
x_source = rng.integers(0, 8, size=n_channel)  # 3 bits of source entropy
y_clean = x_source  # perfect first channel

mi_xy = mutual_information(count_pairs(x_source, y_clean, 8, 8))

noise_levels = range(0, 7)
mi_curve = []
for half_width in noise_levels:
    noise = rng.integers(-half_width, half_width + 1, size=n_channel)
    z_noisy = np.clip(x_source + noise, 0, 15)
    mi_curve.append(mutual_information(count_pairs(x_source, z_noisy, 8, 16)))
mi_curve = np.array(mi_curve)

Out[17]:

Console

I(X ; Y) with a clean channel:          3.000 bits  (source entropy = 3)
I(X ; Z) with noise in [-+0, +0]:  3.000 bits
I(X ; Z) with noise in [-+1, +1]:  1.613 bits
I(X ; Z) with noise in [-+2, +2]:  1.053 bits
I(X ; Z) with noise in [-+3, +3]:  0.735 bits
I(X ; Z) with noise in [-+4, +4]:  0.543 bits
I(X ; Z) with noise in [-+5, +5]:  0.414 bits
I(X ; Z) with noise in [-+6, +6]:  0.319 bits

The clean channel preserves all three bits. Every noisy variant destroys some, and the loss grows monotonically with the noise width, exactly as the data processing inequality demands. Notice also what the numbers encode: with moderate noise, $Z$ retains a bit or two about $X$ , meaning a receiver can still partially infer the source, but no downstream processing of $Z$ can recover the missing bits. Partial preservation is the realistic regime for world models: a lossy encoder leaves the latent partially informative about the past, and the art lies in ensuring the surviving fraction covers what the future will ask about.

Out[18]:

Visualization

Line chart of mutual information decreasing from 3 bits toward zero as noise widens. — Mutual information between the source and the channel output decays monotonically as channel noise grows, illustrating the data processing inequality.

Sufficient statisticsLink Copied

The data processing inequality tells us compression loses information in general. Sufficiency characterizes the compression that loses nothing relevant.

Sufficient Statistic

A function $T$ of data $x$ is a sufficient statistic for a parameter $\theta$ if the data add no information about $\theta$ beyond $T(x)$ :

p(x \mid T(x), \theta) \text{ does not depend on } \theta.

Equivalently, in information terms, $I(X; \theta) = I(T(X); \theta)$ : the compressed summary preserves all information relevant to the question being asked, while discarding everything irrelevant.

The classic example: to estimate the mean of a Gaussian, the sample mean is sufficient; the individual orderings and deviations of the data points carry no additional information about that particular question. Sufficiency is always relative to a purpose. A summary sufficient for one question is usually wildly insufficient for another. The sample mean of a Gaussian tells you nothing about its variance; the running maximum of a river gauge is sufficient for flood risk but nearly useless for predicting next week's level. This relativity is not a weakness but the entire point: it converts the vague demand "keep everything important" into the precise demand "keep everything important for the specified question", and for world models the specified question is always prediction of the future.

From sufficient statistics to sufficient statesLink Copied

Now assemble the pieces, because they snap together into the defining property of a good world-model state. Recall the Markov property from earlier: the future is conditionally independent of the past given the present state. Restate that in information-theoretic language:

I(\text{future}; \text{past} \mid s_t) = 0.

The state $s_t$ is a sufficient statistic of the history for the purpose of predicting the future. Everything in the past that matters has been routed through the state; everything discarded was irrelevant. This single sentence connects three threads of this handbook:

Filtering: when observations are partial, the belief state maintained by a Bayesian filter is designed to be exactly such a sufficient summary, and we will construct it in Part II: Inference, Decisions, and Control.
State abstraction: when we search for compact states that preserve decision-relevant information while discarding the rest, we are searching for approximate sufficient statistics, the topic of Part III's treatment of state abstraction and sufficient statistics.
Representation learning: when a latent-variable model is trained so that predictions depend on the past only through the latent, the data processing inequality certifies what the encoder is allowed to forget, a constraint we exploit in Part III's treatment of latent representations.

Approximation enters inevitably. Exact sufficiency usually requires carrying enormous state; practical world models trade a controlled amount of predictive information for a large reduction in state size. Much of the art of world modeling is choosing where along that trade-off curve to sit, and we will develop tools for judging that choice in Part XI. The thermostat below offers a rare treat: a system where the sufficient state is exactly one bit larger than the obvious choice, so we can see the cost of insufficiency in closed form rather than in vague degradation.