Modern Portfolio Theory: Mean-Variance Optimization Guide

In 1952, Harry Markowitz published a paper titled "Portfolio Selection" that fundamentally changed how investors think about risk and return. Before Markowitz, investment analysis focused almost exclusively on picking individual securities with the best expected returns. Markowitz introduced a revolutionary insight: the risk of a portfolio depends not just on the risks of individual assets, but critically on how those assets move together. This observation gave birth to Modern Portfolio Theory (MPT) and provided the first rigorous mathematical framework for constructing optimal portfolios.

The core idea is straightforward. Investors care about two things: maximizing expected return and minimizing risk. Markowitz formalized risk as the variance (or standard deviation) of portfolio returns, creating what we now call the mean-variance framework. Within this framework, rational investors should only hold portfolios that offer the highest expected return for a given level of risk, or equivalently, the lowest risk for a given expected return. These optimal portfolios form the efficient frontier, a concept that remains central to investment management today.

This chapter develops the mathematical machinery of mean-variance optimization from first principles. We'll see how diversification arises naturally from the mathematics of combining assets with less-than-perfect correlation, and we'll implement practical algorithms for finding optimal portfolios. The framework we develop here provides the foundation for the Capital Asset Pricing Model and factor models that follow in subsequent chapters.

Before we can optimize portfolios, we need precise definitions of portfolio return and risk. These definitions form the foundation for Modern Portfolio Theory. We need accurate measures of portfolio return and risk to compare investment strategies and identify optimal allocations.

Suppose we allocate wealth across $n$ assets. The fundamental question is: how should we combine individual asset characteristics to describe the portfolio as a whole? Let $w_i$ denote the fraction of wealth invested in asset $i$, where the weights must sum to one:

where:

• $w_i$: fraction of wealth invested in asset $i$
• $n$: total number of assets in the portfolio

This constraint ensures we account for all invested capital. A weight of 0.3 means 30% of the portfolio is allocated to that asset. We allow short selling, meaning weights can be negative (representing borrowed positions). Shorting an asset involves selling borrowed shares with the obligation to return them later, creating a negative position that profits when the price falls.

The expected return of a portfolio follows naturally from the linearity of expectation, a concept we explored in our probability foundations. If each asset $i$ has expected return $\mu_i = E[R_i]$, the portfolio's expected return is simply the weighted average:

where:

• $\mu_p$: expected return of the portfolio
• $E[R_p]$: expected value of portfolio return
• $w_i$: weight of asset $i$
• $\mu_i$: expected return of asset $i$
• $\mathbf{w}$: vector of portfolio weights, $(w_1, \ldots, w_n)^\top$
• $\boldsymbol{\mu}$: vector of expected returns, $(\mu_1, \ldots, \mu_n)^\top$

This formula tells us something intuitive: a portfolio's expected return is determined entirely by how much we invest in each asset and what return we expect from that asset. There are no interaction effects or nonlinearities. If we double our allocation to a high-return asset, we increase the portfolio's expected return proportionally. Vector notation provides a compact way to represent many assets.

The portfolio variance requires more care and reveals how diversification works mathematically. Unlike expected return, variance does not simply average across assets. The portfolio return is:

where:

• $R_p$: return of the portfolio
• $w_i$: weight of asset $i$
• $R_i$: return of asset $i$
• $n$: number of assets

To find the variance of this weighted sum, we must account for how each pair of assets moves together. Taking the variance and using the linearity properties we covered in Part I:

where:

• $\sigma_p^2$: variance of the portfolio
• $w_i, w_j$: weights allocated to assets $i$ and $j$
• $\text{Cov}(R_i, R_j)$: covariance between returns of assets $i$ and $j$

The double summation captures all pairwise interactions between assets. When $i = j$, the covariance term becomes the variance of asset $i$, contributing $w_i^2 \sigma_i^2$ to portfolio variance. When $i \neq j$, the covariance measures how assets $i$ and $j$ move together, and this is where diversification benefits emerge.

Let $\Sigma$ denote the $n \times n$ covariance matrix, where $\Sigma_{ij} = \text{Cov}(R_i, R_j)$. Then the portfolio variance is:

where:

• $\sigma_p^2$: variance of the portfolio
• $\mathbf{w}$: vector of portfolio weights
• $\Sigma$: covariance matrix of asset returns

This compact matrix notation, which we developed in Part I's linear algebra chapter, makes optimization tractable. The quadratic form $\mathbf{w}^\top \Sigma \mathbf{w}$ appears throughout financial mathematics because it elegantly encapsulates how portfolio risk depends on both individual asset volatilities (the diagonal of $\Sigma$) and their correlations (the off-diagonal elements).

The two-asset case provides crucial intuition before we tackle the general problem. By working through this simplified scenario, we can develop a geometric understanding of how risk and return interact, seeing clearly how correlation shapes the set of achievable portfolios. Consider a portfolio with weights $w$ in asset 1 and $(1-w)$ in asset 2.

The expected return is:

where:

• $\mu_p$: expected return of the portfolio
• $w$: weight invested in asset 1
• $\mu_1, \mu_2$: expected returns of assets 1 and 2

This formula shows that expected return varies linearly with the weight $w$. As we shift allocation from asset 2 toward asset 1, expected return moves along a straight line between $\mu_2$ (when $w=0$) and $\mu_1$ (when $w=1$). There are no surprises here: return blending is straightforward.

The variance, however, tells a richer story:

where:

• $\sigma_p^2$: variance of the portfolio
• $w$: weight invested in asset 1
• $\sigma_1, \sigma_2$: standard deviations (volatility) of assets 1 and 2
• $\rho_{12}$: correlation coefficient between returns of assets 1 and 2

This equation has three terms, each revealing a different aspect of portfolio risk. The first term, $w^2\sigma_1^2$, captures the contribution of asset 1's own volatility, scaled by the square of its weight. The second term, $(1-w)^2\sigma_2^2$, does the same for asset 2. The third term, the cross-term $2w(1-w)\rho_{12}\sigma_1\sigma_2$, is where the diversification benefit arises. This term depends on the correlation $\rho_{12}$, and its sign and magnitude determine whether combining the two assets reduces or amplifies total portfolio risk.

The diversification effect emerges from this variance formula. Consider what happens when $\rho_{12} < 1$. The cross-term $2w(1-w)\rho_{12}\sigma_1\sigma_2$ is smaller than it would be under perfect correlation, reducing total portfolio variance. When correlation is negative, this term actually becomes negative, further reducing portfolio risk below what the individual variances would suggest.

To understand this intuitively, imagine two assets that tend to move in opposite directions. When one zigs, the other zags. In a portfolio containing both, these opposite movements partially cancel each other, smoothing the overall portfolio return and reducing its volatility. The lower the correlation, the more pronounced this cancellation effect.

Let's see this numerically:

The figure reveals several key insights about how correlation shapes investment opportunities:

• Perfect positive correlation (\rho = 1): The frontier is a straight line. Diversification provides no risk reduction; portfolio risk is simply the weighted average of individual risks. The two assets move in perfect lockstep, so combining them cannot smooth out volatility.
• Imperfect correlation ($\rho < 1$): The frontier curves leftward, meaning some portfolios achieve lower risk than either asset alone. This is the diversification benefit. The curvature increases as correlation decreases, opening up more favorable risk-return combinations.
• Perfect negative correlation ($\rho = -1$): A risk-free portfolio exists. When $w = \sigma_2 / (\sigma_1 + \sigma_2)$, the portfolio variance is exactly zero. This extreme case illustrates the theoretical maximum of diversification: if two assets move in perfectly opposite directions, we can combine them to eliminate all uncertainty.

For the two-asset case, we can find the minimum variance portfolio analytically. This portfolio represents the allocation that achieves the lowest possible risk, regardless of the expected return it delivers. To find the minimum variance portfolio, we differentiate the variance $\sigma_p^2$ with respect to $w$ and set the result to zero:

The derivative includes three components corresponding to the three terms in the variance formula. Setting this derivative to zero identifies the point where small changes in $w$ produce no change in variance, which is precisely the minimum. Rearranging the first-order condition to isolate $w$:

Solving for $w$:

where:

• $w^*$: optimal weight for asset 1 to minimize portfolio variance
• $\sigma_1, \sigma_2$: volatilities of assets 1 and 2
• $\rho_{12}$: correlation between assets 1 and 2

This formula balances the volatilities and correlation to minimize risk. If the assets are uncorrelated ($\rho_{12}=0$), the weight $w^*$ is proportional to $\sigma_2^2$, meaning the portfolio holds more of the asset with lower volatility. This makes intuitive sense: when assets don't interact, we should tilt toward the less risky one. As correlation decreases (becomes more negative), the optimal weight shifts to exploit the hedging benefit, potentially allocating more to the higher-volatility asset because its movements offset those of the other asset.

The minimum variance portfolio achieves lower risk than either individual asset. This is the fundamental benefit of diversification. Even though both assets carry uncertainty, combining them in the right proportions produces a portfolio with less uncertainty than holding either one alone. This result, sometimes called the "free lunch" of diversification, arises purely from the mathematics of combining imperfectly correlated random variables.

With more than two assets, we need the full matrix formulation. The principles remain the same, but the algebra becomes more complex, requiring linear algebra tools to express and solve the optimization problem efficiently. Let $\mathbf{w}$ be the $n \times 1$ vector of weights, $\boldsymbol{\mu}$ the $n \times 1$ vector of expected returns, and $\Sigma$ the $n \times n$ covariance matrix.

The optimization problem has two equivalent formulations, each emphasizing a different perspective on the objective:

Formulation 1: Minimize risk for a target return

This formulation starts with a desired level of expected return and asks: what is the least risky way to achieve it?

where:

• $\mathbf{w}$: vector of portfolio weights
• $\Sigma$: covariance matrix of asset returns
• $\boldsymbol{\mu}$: vector of expected returns
• $\mu_{\text{target}}$: required level of expected return
• $\mathbf{1}$: vector of ones (sum constraint)

The factor of $\frac{1}{2}$ in the objective is a mathematical convenience that simplifies the derivatives. It does not change the optimal solution because multiplying an objective by a positive constant does not alter which portfolio minimizes it.

Formulation 2: Maximize utility (mean-variance utility)

This formulation takes a different approach, directly modeling investor preferences through a utility function that trades off return against risk:

where:

• $\gamma$: risk aversion parameter ($\gamma > 0$)
• $\mathbf{w}$: vector of portfolio weights
• $\boldsymbol{\mu}$: vector of expected returns
• $\Sigma$: covariance matrix of asset returns
• $\mathbf{1}$: vector of ones

The parameter $\gamma$ captures how much you dislike risk. If you are highly risk-averse (large $\gamma$), you penalize variance heavily and will choose a conservative, low-volatility portfolio. If you are risk-tolerant (small $\gamma$), you care more about expected return and will accept higher volatility to chase higher gains. Both formulations trace out the same efficient frontier; they simply parameterize it differently.

For the first formulation, we form the Lagrangian using the techniques from Part I. The Lagrangian method converts a constrained optimization problem into an unconstrained one by introducing penalty terms for violating constraints:

where:

• $\mathcal{L}$: the Lagrangian function
• $\mathbf{w}$: vector of portfolio weights
• $\Sigma$: covariance matrix of asset returns
• $\lambda_1, \lambda_2$: Lagrange multipliers enforcing the return and budget constraints
• $\boldsymbol{\mu}$: vector of expected returns
• $\mu_{\text{target}}$: target portfolio return
• $\mathbf{1}$: vector of ones

The Lagrange multipliers $\lambda_1$ and $\lambda_2$ act as shadow prices, measuring how much the optimal objective would change if we relaxed the corresponding constraint slightly. Taking the derivative with respect to $\mathbf{w}$ and setting it to zero:

where:

• $\Sigma$: covariance matrix
• $\mathbf{w}$: vector of portfolio weights
• $\lambda_1, \lambda_2$: Lagrange multipliers
• $\boldsymbol{\mu}$: vector of expected returns
• $\mathbf{1}$: vector of ones

This first-order condition states that at the optimum, the marginal increase in risk from adjusting any weight must be exactly offset by the value of that weight in meeting the return and budget constraints. Rearranging terms to solve for $\mathbf{w}$ explicitly:

where:

• $\mathbf{w}$: optimal portfolio weight vector
• $\Sigma^{-1}$: inverse of the covariance matrix
• $\boldsymbol{\mu}$: vector of expected returns
• $\mathbf{1}$: vector of ones
• $\lambda_1, \lambda_2$: Lagrange multipliers

The formula decomposes the optimal portfolio into two parts: a term proportional to $\Sigma^{-1}\boldsymbol{\mu}$ (seeking high returns) and a term proportional to $\Sigma^{-1}\mathbf{1}$ (minimizing variance). The multipliers $\lambda_1$ and $\lambda_2$ weight these components to satisfy the target return and budget constraints. This decomposition reveals the fundamental tension in portfolio construction: the desire for return pulls toward $\Sigma^{-1}\boldsymbol{\mu}$, while the desire for safety pulls toward $\Sigma^{-1}\mathbf{1}$.

The Lagrange multipliers $\lambda_1$ and $\lambda_2$ are determined by the constraints. Substituting back:

where:

• $\mu_{\text{target}}$: target return
• $\lambda_1, \lambda_2$: Lagrange multipliers
• $\boldsymbol{\mu}$: vector of expected returns
• $\Sigma^{-1}$: inverse covariance matrix
• $\mathbf{1}$: vector of ones

These two equations form a linear system in the two unknowns $\lambda_1$ and $\lambda_2$. We can solve for $\lambda_1$ and $\lambda_2$ in terms of scalar constants derived from the problem parameters:

where:

• $A = \boldsymbol{\mu}^\top \Sigma^{-1} \boldsymbol{\mu}$
• $B = \boldsymbol{\mu}^\top \Sigma^{-1} \mathbf{1} = \mathbf{1}^\top \Sigma^{-1} \boldsymbol{\mu}$
• $C = \mathbf{1}^\top \Sigma^{-1} \mathbf{1}$
• $\mu_{\text{target}}$: target portfolio return

The constants $A$, $B$, and $C$ depend only on the assets' expected returns and covariance structure, not on the target return. This means we can precompute them once and then quickly find optimal portfolios for any target return, making efficient frontier construction computationally efficient.

Setting $\lambda_1 = 0$ (ignoring the return target), we get the global minimum variance (GMV) portfolio:

where:

• $\mathbf{w}_{\text{GMV}}$: weights of the global minimum variance portfolio
• $\Sigma^{-1}$: inverse of the covariance matrix
• $\mathbf{1}$: vector of ones

This formula represents risk minimization in its most basic form. The term $\Sigma^{-1}\mathbf{1}$ identifies the portfolio structure that minimizes variance without regard for expected returns, while the denominator normalizes the weights to sum to one. Assets with lower variance and lower correlation to other assets receive higher weights, as they contribute less to overall portfolio risk.

This portfolio has the lowest possible variance among all portfolios satisfying the budget constraint. It sits at the leftmost point of the efficient frontier, representing the safest allocation available without going to cash. If you care only about minimizing uncertainty (perhaps because you have very short horizons or extreme risk aversion), the GMV portfolio is optimal regardless of expected returns.

The efficient frontier is the set of all portfolios that offer the maximum expected return for each level of risk. Mathematically, it's the upper boundary of the feasible set in risk-return space. This concept is central to Modern Portfolio Theory because it delineates the best possible tradeoffs available to investors.

To understand the efficient frontier, imagine plotting every possible portfolio in risk-return space. The feasible region, bounded by the budget constraint and any position limits, forms a region. Some portfolios lie on the boundary of this region, while others lie in the interior. Interior portfolios are dominated by boundary portfolios that offer either higher return for the same risk or lower risk for the same return. The efficient frontier consists of all portfolios that are not dominated by any other portfolio.

A remarkable property of mean-variance optimization is the two-fund separation theorem: any efficient portfolio can be expressed as a linear combination of two distinct efficient portfolios. This means efficiency can be achieved regardless of risk preferences by holding different combinations of just two mutual funds.

The theorem significantly affects portfolio management. It suggests that an asset management firm need only offer two efficiently managed funds, and an investor can achieve an optimal portfolio by combining these funds in proportions determined by their risk tolerance. This separation between the "production" of efficient portfolios and your "consumption" choices simplifies the investment process dramatically.

Mathematically, if $\mathbf{w}_1$ and $\mathbf{w}_2$ are any two efficient portfolios, then for any scalar $\alpha$, the portfolio $\mathbf{w}$ is also efficient:

where:

• $\mathbf{w}$: the new efficient portfolio
• $\alpha$: weighting scalar
• $\mathbf{w}_1, \mathbf{w}_2$: two distinct efficient portfolios

The parameter $\alpha$ determines where along the efficient frontier the combined portfolio sits. When $\alpha = 1$, we hold only portfolio 1; when $\alpha = 0$, we hold only portfolio 2; values between 0 and 1 produce intermediate portfolios. Values outside this range (which require short selling one of the funds) extend the frontier beyond the original two portfolios.

Let's implement efficient frontier computation using quadratic programming:

Let's construct an efficient frontier for a portfolio of stocks, bonds, and commodities. This example illustrates how different asset classes with distinct risk-return profiles and correlations combine to form a diversified investment opportunity set:

The covariance matrix confirms the relationships between our assets: stocks and commodities have higher volatility and a moderate positive correlation (0.3), while bonds offer stability with low volatility and low correlation to the other assets. The near-zero correlations between bonds and the other asset classes suggest that bonds will play a crucial role in diversifying portfolio risk.

Now let's compute and visualize the efficient frontier:

The efficient frontier curve illustrates the optimal risk-return combinations available to investors. The individual assets (red dots) lie inside the frontier, confirming that portfolios combining imperfectly correlated assets offer superior efficiency compared to holding single securities. Notice how the frontier bows outward to the left, demonstrating that diversification creates portfolios with better risk-return characteristics than any individual asset.

The GMV portfolio is heavily weighted toward bonds because of their low volatility. Notice that the GMV portfolio achieves lower risk than any individual asset, demonstrating the power of diversification. Even the lowest-risk asset (bonds at 5% volatility) has higher risk than the optimally diversified GMV portfolio.

Understanding how portfolio weights change along the efficient frontier provides insight into the risk-return tradeoff. As you move up the frontier seeking higher returns, you must accept more risk, and the portfolio composition shifts accordingly:

At low target returns (near the GMV portfolio), the allocation is dominated by bonds. As we move up the frontier toward higher returns, the portfolio shifts toward stocks, with commodities providing additional diversification benefits due to their low correlation with bonds. The smooth transition in weights illustrates how the efficient allocation changes continuously as your return target increases.

For practical applications with many assets and constraints, we use quadratic programming (QP). Quadratic programming is a mathematical optimization technique designed specifically for problems where the objective function is quadratic (involving squared terms) and the constraints are linear. Mean-variance optimization fits this structure perfectly because portfolio variance is a quadratic function of the weights.

The standard QP form is:

where:

• $\mathbf{x}$: vector of decision variables
• $Q$: symmetric matrix (quadratic term)
• $\mathbf{c}$: vector (linear term)
• $A_{\text{eq}}, \mathbf{b}_{\text{eq}}$: matrix and vector defining equality constraints
• $A_{\text{ineq}}, \mathbf{b}_{\text{ineq}}$: matrix and vector defining inequality constraints

Mean-variance optimization maps directly to this form with $Q = \Sigma$ and $\mathbf{c} = \mathbf{0}$ (or $\mathbf{c} = -\gamma\boldsymbol{\mu}$ for the utility maximization variant). The equality constraints enforce the budget constraint and target return, while inequality constraints can encode position limits, sector constraints, and other practical requirements.

Real portfolio optimization often includes additional constraints that reflect regulatory requirements, risk management policies, or investment mandates:

• No short selling: $w_i \geq 0$ for all $i$. This constraint prevents the portfolio from taking negative positions, which an investor cannot or chooses not to take.
• Maximum position size: $w_i \leq w_{\max}$ (e.g., 10% per asset). This ensures diversification by preventing excessive concentration in any single security.
• Sector constraints: Total exposure to a sector $\leq$ some limit. This prevents the portfolio from becoming overly exposed to a single industry or market segment.
• Turnover constraints: Changes from current portfolio are limited. This controls transaction costs and tax consequences by restricting how much trading occurs.

Let's implement a more realistic optimizer using the cvxpy library:

The constrained portfolios have slightly higher risk than the unconstrained GMV portfolio. This is a general result: adding constraints can never improve the optimal objective value. Constraints restrict the feasible region, preventing the optimizer from reaching solutions that might otherwise be optimal. The cost of these constraints appears as slightly higher portfolio risk, but this cost may be acceptable given the practical benefits of avoiding short positions or maintaining diversification.

Diversification is the only "free lunch" in finance. To understand why, let's decompose portfolio variance more carefully. The key insight is that not all risk is equal: some risk affects all assets together (systematic risk), while other risk is unique to individual securities (idiosyncratic risk). Diversification can eliminate the latter but not the former.

Consider an equally-weighted portfolio of $n$ assets with identical variance $\sigma^2$ and pairwise correlation $\rho$. While this is a simplification, it reveals the essential mechanics of diversification. The portfolio variance is:

where:

• $\sigma_p^2$: variance of the portfolio
• $n$: number of assets
• $\sigma_{ij}$: covariance between asset $i$ and asset $j$

For the diagonal terms ($i = j$): $\sigma_{ii} = \sigma^2$

For the off-diagonal terms ($i \neq j$): $\sigma_{ij} = \rho \sigma^2$

There are $n$ diagonal terms and $n(n-1)$ off-diagonal terms. The diagonal terms represent each asset's own variance contribution, while the off-diagonal terms capture the covariance contributions from all pairs of different assets:

where:

• $\sigma^2$: variance of each individual asset
• $\rho$: pairwise correlation between all assets
• $n$: number of assets in the portfolio

This decomposition separates portfolio variance into two distinct components. The first term, $\sigma^2/n$, represents idiosyncratic risk. It depends on the number of assets and shrinks as $n$ increases. This is the diversifiable component of risk. The second term, $\frac{n-1}{n}\rho\sigma^2$, represents systematic risk. It depends on the correlation between assets and persists regardless of how many assets we hold.

As $n \to \infty$:

where:

• $\sigma_p^2$: limiting portfolio variance
• $\rho$: pairwise correlation between assets
• $\sigma^2$: variance of individual assets

This reveals a fundamental insight: diversification can eliminate idiosyncratic risk (the $\sigma^2/n$ term that vanishes with $n$), but cannot eliminate systematic risk (the $\rho \sigma^2$ term that persists). No matter how many assets we add to our portfolio, we cannot diversify away the risk that comes from common factors affecting all assets simultaneously. When markets crash, all stocks tend to fall together; this correlated movement represents systematic risk that no amount of stock picking can eliminate.

The figure shows that most diversification benefits are captured with 20-30 assets, depending on correlation. Beyond that, adding assets provides diminishing marginal benefit. This explains why well-diversified portfolios don't need hundreds of positions. The mathematics tells us that each additional asset beyond a certain point contributes very little to further risk reduction, while potentially adding complexity and transaction costs.

The table results demonstrate that the majority of diversification benefits are realized early. With just 20 assets, nearly 95% of the diversifiable risk is eliminated, confirming that a portfolio does not need hundreds of positions to be well-diversified. This finding has practical implications for portfolio construction: beyond a certain point, adding more securities increases complexity and costs without meaningfully improving the risk-return profile.

Let's apply mean-variance optimization to a realistic multi-asset portfolio using historical return data: