Perceptual and Generative Evaluation › Reconstruction and Predictive Accuracy

Reconstruction scoring and future-prediction scoring agree on the math and disagree on the story. In reconstruction the model sees $o_t$, produces $\hat o_t$, and we compare $\hat o_t$ to $o_t$. In future prediction the model sees $h_t$ and $a_{t:t+H-1}$ and produces $\hat o_{t+1:t+H}$, and we compare the predicted sequence to the realized sequence $o_{t+1:t+H}$. Pixel-wise mean squared error (MSE) and peak signal-to-noise ratio (PSNR) are the two workhorses of both. Frame MSE averages squared pixel residuals; frame PSNR applies a logarithm to that spatially averaged MSE. For sequences, averaging frame PSNR is not the same as applying the logarithm to a pooled MSE. The averaging conventions matter and are the subject of the next formulas.

The reference changes meaning between tasks. Ordinary reconstruction compares against the fixed input observation; zero pixel MSE requires reproducing that array, but reconstruction quality alone does not establish useful internal state. Future prediction compares against a realized outcome, potentially one of several possible futures. The same distance therefore has a different interpretation.

The mean squared error (MSE) between a prediction $\hat x$ and a target $x$, both of shape $(H_{\mathrm{img}}, W)$ with pixel values in $[0,1]$, is

where:

• $\hat x_{ij}$: the predicted pixel value at spatial position $(i, j)$
• $x_{ij}$: the target pixel value at the same position
• $H_{\mathrm{img}}, W$: the height and width of the frame in pixels
• $\frac{1}{H_{\mathrm{img}}W}$: normalization by the pixel count, making the result an average rather than a sum

The subtraction $\hat x_{ij} - x_{ij}$ is the signed per-pixel error. Squaring removes its sign and heavily penalizes large deviations.

The square grows quadratically with error magnitude. Uniform error of 0.1 gives MSE 0.01. Error of 0.3 on 10 percent of pixels, with zero error elsewhere, gives MSE $0.09\times0.1=0.009$. These close averages do not describe where the errors occur. The perceptual importance of a localized error depends on the scene and viewing task; we have not measured it in this calculation.

The quadratic penalty is easiest to see as a curve. The same squared-error contribution can come from a small error everywhere or a larger error in a small region.

For a multi-step prediction, we first compute the per-frame MSE for each horizon step and then average those per-frame values over time. The sequence MSE is

where:

• $\hat o_{t+k}$: the predicted frame at horizon step $k$
• $o_{t+k}$: the realized frame at the same horizon step
• $H$: the number of future frames being averaged
• $\mathrm{MSE}(\hat o_{t+k}, o_{t+k})$: the per-frame MSE defined above

Notice the two averaging decisions: how to combine spatial and channel dimensions into a per-frame value, and how to combine frames into a per-rollout value. Do not compare two numbers from different conventions. Always specify the pixel range (here in $[0,1]$) and the tensor layout, and state whether frames of different length are padded and excluded via a valid mask $m_k \in \{0, 1\}$:

where:

• $m_k \in \{0, 1\}$: a validity indicator for frame $k$, where $m_k = 1$ includes the frame and $m_k = 0$ excludes it (e.g., padding)
• $H$: the prediction horizon, as before
• $\sum_{k=1}^{H} m_k$: the valid-frame count, which must be positive. An all-padding sequence has no defined score under this formula and must be excluded or handled by a declared convention.

The valid mask is not a cosmetic detail. Suppose a model is trained to predict up to 64 frames, but two datasets provide rollouts of length 32 and 48 respectively. If the evaluation pads all rollouts to length 64 and averages the padded MSE without masking, the shorter dataset will be artificially penalized or rewarded depending on what value padded frames take. A mask removes this artifact by restricting the average to the frames the data actually contains. The same averaging convention question also arises when sequences are grouped into batches: the natural convention is to compute a per-sequence score and then average over sequences, rather than pooling all frames across the batch, because the latter weights long sequences more heavily.

Peak signal-to-noise ratio (PSNR) is MSE expressed on a logarithmic scale. It answers the question: how large is the squared peak signal relative to the squared reconstruction error? Given a declared peak value $L$ (usually the maximum representable pixel value, so $L = 1$ for a $[0, 1]$ range and $L = 255$ for 8-bit images), PSNR is

where:

• $L$: the declared peak value of the pixel range, i.e. the maximum representable pixel value
• $\mathrm{MSE}(\hat x, x)$: the pixel MSE between prediction and target, defined above
• $10 \log_{10}(\cdot)$: a logarithmic rescaling of the ratio into decibels (dB). At fixed $L$, lower MSE means higher PSNR, so the preference ordering is equivalent with opposite score directions.

When $\mathrm{MSE} = 0$, PSNR is $+\infty$ in the mathematical definition. If a report clips exact matches to a finite cap, it must state that choice. Declare $L$ and keep it consistent with the pixel units. Rescaling both images by 255 multiplies MSE by $255^2$ and changes $L$ from 1 to 255, leaving PSNR unchanged. Changing only $L$ while keeping the same numerical errors adds $20\log_{10}(255) \approx 48.13$ dB, an inconsistent protocol rather than a model improvement. At fixed $L$, PSNR is a strictly convex function of positive MSE: a constant minus $10\log_{10}(\mathrm{MSE})$.

For positive frame errors $e_1,\ldots,e_n$ and the same $L$, Jensen's inequality gives mean frame PSNR greater than or equal to PSNR of mean MSE, with equality when all errors agree. Their gap is $10\log_{10}(\mathrm{AM}(e)/\mathrm{GM}(e))$, the logarithm of the arithmetic-to-geometric mean ratio. It increases under a mean-preserving spread of the errors; variance alone does not determine it. State which averaging convention the table uses. Exact-zero errors need an explicit infinity or clipping convention.

The next bar chart applies both averaging conventions to the same eight positive frame errors. Its difference measures the averaging gap for this fixed set, not a comparison between two predictors.

Expected MSE is minimized by the conditional mean. Suppose $y$ is a real-valued target with finite second moment, $x$ contains the conditioning information, and $f(x)$ is an unrestricted measurable predictor with finite squared error. The expectation is over the joint distribution $p(x,y)$. The squared error decomposes into an irreducible term plus the squared gap between $f$ and the conditional mean:

Expanding the square and conditioning on $x$ eliminates the cross term: the conditional mean of $y-\mathbb{E}[y\mid x]$ is zero almost surely. The irreducible first term does not depend on $f$; the second is nonnegative and is zero exactly when $f(x)=\mathbb{E}[y\mid x]$ almost surely. Thus the conditional mean minimizes expected squared error under the stated integrability assumptions.

For a finite-dimensional image vector, apply the same decomposition to each coordinate and average. The conditional mean still minimizes expected pixel MSE. If possible images place an object at separated locations, averaging their pixels can produce a faded overlay rather than an image from either mode. This is a possible consequence of the geometry, not a claim that every multimodal distribution has an implausible mean.

In our separated-object example, an MSE-optimal point estimate need not be a plausible sample. That is a mismatch between the prediction target and the sample-quality question, not a contradiction in the theorem. Blur can also reflect limited decoder resolution or capacity; the image alone does not identify its cause.

Structural similarity (SSIM), introduced by Wang, Bovik, Sheikh, and Simoncelli (2004), compares local luminance, contrast, and structure rather than uniformly weighting pixel-coordinate squared errors. For each sliding-window position, it summarizes corresponding patches by weighted means, variances, and covariance. With normalized Gaussian window weights $w$, these statistics are defined as:

where:

• $w_i$: the window weight at position $i$ (Gaussian or uniform), summing to 1
• $x_i, y_i$: pixel values of the two images within the window
• $\mu_x, \mu_y$: the weighted local means
• $\sigma_x^2, \sigma_y^2$: the weighted local variances
• $\sigma_{xy}$: the weighted local covariance between the two patches, computed as $\sum_i w_i (x_i - \mu_x)(y_i - \mu_y)$

The SSIM score for that window is

where:

• $(2 \mu_x \mu_y + C_1)$: the numerator of the luminance term, comparing the two local means
• $(\mu_x^2 + \mu_y^2 + C_1)$: the denominator of the luminance term, normalizing that comparison
• $(2 \sigma_{xy} + C_2)$: the numerator of the combined contrast-and-structure term
• $(\sigma_x^2 + \sigma_y^2 + C_2)$: its denominator; together the factors compare covariance and local variance, not a squared correlation alone
• $C_1 = (K_1 L)^2$, $C_2 = (K_2 L)^2$: small stabilizing constants using the reference defaults $K_1 = 0.01$ and $K_2 = 0.03$, where $L$ is the declared dynamic range

The first ratio measures luminance agreement and the second measures structural (contrast and correlation) agreement; the constants $C_1, C_2$ keep the ratio finite when the local variances or means are near zero. The per-window scores are averaged over all window positions, giving the mean SSIM (MSSIM) that is reported for image-quality benchmarking.

The second ratio is not squared Pearson correlation. Without stabilizers and with positive local standard deviations, it equals the correlation multiplied by $2\sigma_x\sigma_y/(\sigma_x^2+\sigma_y^2)$, a contrast-agreement factor. A positive gain can preserve correlation while reducing contrast agreement; an offset can reduce luminance agreement. SSIM does not ignore brightness shifts. Rescaling both images and the declared range $L$ together preserves this formula, but that change of units is not invariance to a physical gain or offset applied to just one image.

SSIM depends on the window size, whether the window is Gaussian or uniform, the standard deviation of the Gaussian, border-handling, whether the input is color or grayscale, and the declared $L$. Two papers calling their metric "SSIM" may be computing different numbers. And, more importantly, SSIM is a proxy for human similarity judgments on natural images; it is not a semantic or physical truth classifier. A prediction with high SSIM can still place an object on the wrong side of a wall.

SSIM's agreement with human ratings was evaluated on specified image-distortion datasets. Agreement on a new domain or task is not guaranteed, but domain change does not necessarily reduce it. Validate the score against the distinctions that matter for the target task.

Consider a grayscale $48 \times 48$ canvas with a Gaussian blob of width $\sigma=2.5$ pixels. We prescribe two equally likely future locations under the same action, which could represent unobserved friction or contact variation. The code renders the two endpoints; it does not simulate that physical mechanism. The realized reference is the short-step endpoint. We compare a sharp match, a sharp alternative, and their pixel-wise mean using MSE and PSNR. At a fixed range on a single pair, PSNR gives exactly the reverse numerical ordering of MSE, so their preferences agree. We discuss SSIM separately rather than pretending to have computed it here.

The soft rendering makes the pixel-wise average visible as a faded superposition. Gaussian blobs have overlapping tails, so they are separated rather than strictly disjoint. This particular average is neither sharp endpoint; the conditional-mean theorem does not assert that every mean of separated modes lies in a low-density region.

Now we build a single realized future and three candidate predictions. The realized future is mode A. The candidates are mode A (Sharp-A), mode B (Sharp-B), and the pixel-wise conditional mean. The mean under a two-mode conditional with equal weights is the average of the two mode images.

Now score the predictions against the realized future. The mean averages two separated blobs with overlapping tails. Its maximum intensity is approximately 0.503 in this discretized example, close to half the peak of either endpoint, so it looks faded.

The bar chart makes the first-class concern of this chapter concrete. Pixel error rewards the model that got lucky on this particular draw, not the model that captured the conditional. Sharp-B is a mode the ground-truth process itself can produce; if the realized future had been Sharp-B instead, the ranking would flip. And the conditional mean, which is MSE-optimal on average, looks neither like Sharp-A nor Sharp-B in this particular draw.

One paired score evaluates one prediction against one realized future. Repeated paired MSE estimates expected squared error, but even that expectation does not characterize the full conditional distribution: the conditional mean is optimal without representing its modes or probabilities. Distributional tests with explicit conditioning or proper probabilistic scores answer a different question.

MSE remains useful for measuring paired accuracy. To assess the forecast distribution, use a declared probabilistic scoring rule or conditional distributional diagnostic over enough held-out cases. Merely averaging more paired MSE values does not turn MSE into a complete distributional test.