Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Advanced Portfolio Construction Techniques › The Problem with Mean-Variance Optimization

Michael BrenndoerferDecember 18, 20255 min read

Quantitative FinanceAll chapters

Part 9 of 15Portfolio Theory and Investment Analysis

Chapter 51Advanced Portfolio Construction Techniques › The Problem with Mean-Variance Optimization

Reading section 2 of 11 · The Problem with Mean-Variance Optimization · 5 minSection 2 of 11 · 5 min

Section 2 of 11 · 5 min read

Advanced Portfolio Construction Techniques › The Problem with Mean-Variance OptimizationOne section · Sections in this chapter2 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before diving into solutions, we must first understand why traditional MVO fails in practice. The problem is not conceptual but statistical: we are asking the optimizer to work with inputs that we cannot measure precisely. Recall from Part IV, Chapter 1 that mean-variance optimization finds the portfolio weights $\mathbf{w}$ that minimize portfolio variance for a given target return:

\begin{aligned} & \min_{\mathbf{w}} \frac{1}{2} \mathbf{w}^T \boldsymbol{\Sigma} \mathbf{w} \\ & \text{subject to:} \\ & \mathbf{w}^T \boldsymbol{\mu} = \mu_p \\ & \mathbf{w}^T \mathbf{1} = 1 \end{aligned}

where:

$\mathbf{w}$ : vector of portfolio weights
$\boldsymbol{\Sigma}$ : asset covariance matrix
$\boldsymbol{\mu}$ : vector of expected asset returns
$\mu_p$ : target portfolio return
$\mathbf{1}$ : vector of ones (sum of weights constraint)

The mathematical elegance of this formulation masks a fundamental challenge: every input to this optimization must be estimated from historical data, and even small estimation errors can produce dramatically different optimal portfolios. To understand why, consider what the optimizer is doing. It searches for the portfolio that achieves the target return with minimum variance, which means it naturally gravitates toward assets that appear to have high returns and low volatilities, while avoiding assets that seem risky or poorly performing. The problem is that "appear to have" and "seem" are doing a lot of work in that sentence. An asset that happened to perform well in our historical sample might have been lucky rather than fundamentally superior. An asset with low measured volatility might simply have experienced a calm period. The optimizer cannot distinguish signal from noise, so it treats every fluctuation in our estimates as real information to be exploited.

In[2]:

Code

import numpy as np
from scipy.optimize import minimize

np.random.seed(42)

# True parameters for 5 assets
assets = ["Asset A", "Asset B", "Asset C", "Asset D", "Asset E"]
true_mu = np.array([0.08, 0.10, 0.12, 0.09, 0.11])
true_sigma = np.array([0.15, 0.20, 0.25, 0.18, 0.22])

# Correlation matrix
true_corr = np.array(
    [
        [1.0, 0.3, 0.2, 0.4, 0.3],
        [0.3, 1.0, 0.5, 0.3, 0.4],
        [0.2, 0.5, 1.0, 0.2, 0.5],
        [0.4, 0.3, 0.2, 1.0, 0.3],
        [0.3, 0.4, 0.5, 0.3, 1.0],
    ]
)
true_cov = np.outer(true_sigma, true_sigma) * true_corr


def mvo_weights(mu, cov, target_return=0.10):
    """Compute MVO weights for a given target return."""
    n = len(mu)

    def portfolio_variance(w):
        return w @ cov @ w

    constraints = [
        {"type": "eq", "fun": lambda w: np.sum(w) - 1},
        {"type": "eq", "fun": lambda w: w @ mu - target_return},
    ]

    result = minimize(
        portfolio_variance,
        np.ones(n) / n,
        method="SLSQP",
        constraints=constraints,
    )
    return result.x


# Simulate estimation from different sample sizes
sample_sizes = [60, 120, 252, 504]  # Monthly observations
n_simulations = 100

weight_variance = []
for T in sample_sizes:
    weights_list = []
    for _ in range(n_simulations):
        # Generate returns from true distribution
        returns = np.random.multivariate_normal(true_mu / 12, true_cov / 12, T)
        est_mu = returns.mean(axis=0) * 12
        est_cov = np.cov(returns.T) * 12

        try:
            w = mvo_weights(est_mu, est_cov)
            weights_list.append(w)
        except:
            continue

    weights_array = np.array(weights_list)
    weight_variance.append(weights_array.std(axis=0).mean())

Out[3]:

Console

Average standard deviation of optimal weights across simulations:
-------------------------------------------------------
Sample size  60 months: 0.310
Sample size 120 months: 0.228
Sample size 252 months: 0.233
Sample size 504 months: 0.226

The results reveal a troubling sensitivity: even with several years of data, the "optimal" weights fluctuate dramatically depending on which particular sample we observe. With five years of monthly data (60 observations), the standard deviation of optimal weights across simulations is substantial. Even with over 40 years of data (504 observations), the instability remains problematic. This instability arises because MVO treats the estimated parameters as if they were known with certainty, when in fact they are noisy estimates of unknown true values. The optimizer has no mechanism to express doubt or hedge against the possibility that its inputs are wrong.

Out[4]:

Visualization

Bar chart showing average standard deviation of optimal portfolio weights for four sample sizes: 0.310 at 5 years, 0.228 at 10 years, 0.233 at 21 years, and 0.226 at 42 years. Instability generally falls with more observations but remains material. — Average standard deviation of optimal portfolio weights across varying sample sizes. Weight instability generally falls as the sample size grows but remains substantial even with decades of observations, illustrating how estimation uncertainty translates into portfolio instability.

Estimation Error Amplification

Mean-variance optimization acts as an "error maximizer" because it assigns the largest weights to assets with the highest estimated expected returns and lowest estimated variances. Unfortunately, these extremes are often due to estimation error rather than true superior risk-adjusted returns. An asset that happened to have unusually good returns in our sample will receive a large positive weight, while an asset that happened to have poor returns will be shorted heavily. The optimizer is not wrong given its inputs; the problem is that its inputs are unreliable.