Probability Basics › Expected Value and Variance

While a full probability distribution gives complete information about a random variable, we often need summary statistics that capture the key features with just a few numbers. The two most important summary measures are the expected value and variance, which describe the center and spread of a distribution.

The expected value (or mean) of a random variable represents its long-run average value if we repeated the random process infinitely many times. For a discrete random variable $X$ with values $x_1, x_2, \ldots, x_n$ and corresponding probabilities $p_1, p_2, \ldots, p_n$, the expected value is:

$$E[X] = \sum_{i=1}^n x_i \cdot p_i$$

Example: Expected Number of Heads in Coin Flips

For a binomial random variable (like counting heads in coin flips), there's a convenient formula: $E[X] = n \cdot p$, where $n$ is the number of trials and $p$ is the probability of success on each trial. If we flip a fair coin 100 times, the expected number of heads is:

$$E[X] = 100 \times 0.5 = 50$$

This means that over many repetitions of 1000 coin flips, the average number of heads will converge to 500.

Example: Expected Value of a Fair Die Roll

Consider rolling a fair six-sided die. The random variable $X$ represents the outcome, which can be any value from 1 to 6, each with probability $\frac{1}{6}$:

$$E[X] = 1 \cdot \frac{1}{6} + 2 \cdot \frac{1}{6} + 3 \cdot \frac{1}{6} + 4 \cdot \frac{1}{6} + 5 \cdot \frac{1}{6} + 6 \cdot \frac{1}{6} = \frac{21}{6} = 3.5$$

Notice that the expected value of 3.5 is not even a possible outcome of a single die roll! This illustrates an important point: the expected value represents the long-run average, not necessarily an achievable result. If you rolled a die 1000 times and averaged all the results, you'd get very close to 3.5.

For a continuous random variable with probability density function $f(x)$, the expected value is:

$$E[X] = \int_{-\infty}^{\infty} x \cdot f(x) \, dx$$

The expected value provides a measure of central tendency, answering the question: "What value should I expect on average?" However, knowing only the average doesn't tell us how much variation to expect around that average.

Example: Expected Value of an Exponential Distribution

Consider a continuous random variable $X$ following an exponential distribution with rate parameter $\lambda = 0.5$. The exponential distribution models waiting times between events (like time between customer arrivals or component failures). Its probability density function is:

$$f(x) = \lambda e^{-\lambda x} \quad \text{for } x \geq 0$$

To find the expected value, we compute the integral of $x$ weighted by the probability density function:

$$E[X] = \int_0^{\infty} x \cdot f(x) \, dx = \int_0^{\infty} x \cdot \lambda e^{-\lambda x} \, dx$$

This integral requires integration by parts, which follows the formula:

$$\int u \, dv = uv - \int v \, du$$

We strategically choose our parts to simplify the integral. The key insight is to let $u$ be the polynomial term (which simplifies when differentiated) and $dv$ be the exponential term (which remains manageable when integrated):

$$\begin{aligned} u &= x & \Rightarrow && du &= dx \\ dv &= \lambda e^{-\lambda x} \, dx & \Rightarrow && v &= -e^{-\lambda x} \end{aligned}$$

To verify the calculation of $v$: when we integrate $\lambda e^{-\lambda x}$, we use substitution. Let $w = -\lambda x$, so $dw = -\lambda \, dx$, which means $dx = -\frac{1}{\lambda} dw$. Then:

$$\begin{aligned} \int \lambda e^{-\lambda x} \, dx &= \int \lambda e^{w} \cdot \left(-\frac{1}{\lambda}\right) dw \\ &= -\int e^{w} \, dw \\ &= -e^{w} \\ &= -e^{-\lambda x} \end{aligned}$$

Now we apply integration by parts:

$$\begin{aligned} E[X] &= \int_0^{\infty} x \cdot \lambda e^{-\lambda x} \, dx \\[1em] &= \left[u \cdot v\right]_0^{\infty} - \int_0^{\infty} v \, du \\[1em] &= \left[x \cdot \left(-e^{-\lambda x}\right)\right]_0^{\infty} - \int_0^{\infty} \left(-e^{-\lambda x}\right) \, dx \\[1em] &= \left[-x e^{-\lambda x}\right]_0^{\infty} + \int_0^{\infty} e^{-\lambda x} \, dx \end{aligned}$$

Evaluating the boundary term $\left[-x e^{-\lambda x}\right]_0^{\infty}$:

At the lower limit ($x = 0$):

$$-0 \cdot e^{-\lambda \cdot 0} = -0 \cdot e^{0} = -0 \cdot 1 = 0$$

At the upper limit ($x \to \infty$): We need to evaluate $\lim_{x \to \infty} -x e^{-\lambda x}$. This is an indeterminate form of type $\infty \cdot 0$, where the linear term $x$ grows without bound while the exponential term $e^{-\lambda x}$ decays to zero. To resolve this, we rewrite it as a fraction and apply L'Hôpital's rule:

$$\lim_{x \to \infty} -x e^{-\lambda x} = \lim_{x \to \infty} \frac{-x}{e^{\lambda x}}$$

This is now an $\frac{\infty}{\infty}$ form, so we differentiate the numerator and denominator:

$$\lim_{x \to \infty} \frac{-x}{e^{\lambda x}} = \lim_{x \to \infty} \frac{\frac{d}{dx}(-x)}{\frac{d}{dx}(e^{\lambda x})} = \lim_{x \to \infty} \frac{-1}{\lambda e^{\lambda x}} = \frac{-1}{\infty} = 0$$

The exponential growth in the denominator dominates the linear growth in the numerator. Therefore:

$$\left[-x e^{-\lambda x}\right]_0^{\infty} = 0 - 0 = 0$$

Evaluating the remaining integral $\int_0^{\infty} e^{-\lambda x} \, dx$:

Using the antiderivative $\int e^{-\lambda x} \, dx = -\frac{1}{\lambda} e^{-\lambda x}$ (which can be verified by differentiation):

$$\begin{aligned} \int_0^{\infty} e^{-\lambda x} \, dx &= \left[-\frac{1}{\lambda} e^{-\lambda x}\right]_0^{\infty} \\[1em] &= \lim_{x \to \infty}\left(-\frac{1}{\lambda} e^{-\lambda x}\right) - \left(-\frac{1}{\lambda} e^{-\lambda \cdot 0}\right) \\[1em] &= \left(-\frac{1}{\lambda} \cdot 0\right) - \left(-\frac{1}{\lambda} \cdot 1\right) \\[1em] &= 0 + \frac{1}{\lambda} \\[1em] &= \frac{1}{\lambda} \end{aligned}$$

Note: As $x \to \infty$, we have $e^{-\lambda x} \to 0$ since $\lambda > 0$.

Combining the results:

$$E[X] = 0 + \frac{1}{\lambda} = \frac{1}{\lambda}$$

For $\lambda = 0.5$, this gives $E[X] = \frac{1}{0.5} = 2$. This means that on average, we expect to wait 2 time units between events. The result is intuitive: a higher rate $\lambda$ (more frequent events) leads to a shorter expected waiting time, while a lower rate leads to longer waits. The integral converges beciance

The variance measures how spread out the distribution is around its expected value. It quantifies the average squared deviation from the mean, giving us a sense of how much uncertainty or variability exists in the random variable. The variance of a random variable $X$ is defined as:

$$\text{Var}(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2$$

Where:

• $\text{Var}(X)$ is the variance of the random variable $X$
• $E[X]$ is the expected value (mean)
• $E[X^2]$ is the expected value of the squared random variable

A closely related measure is the standard deviation, which is simply the square root of the variance: $\sigma = \sqrt{\text{Var}(X)}$. The standard deviation has the advantage of being in the same units as the original random variable, making it more interpretable. For example, if we're measuring heights in centimeters, the standard deviation is also in centimeters, while the variance would be in squared centimeters.

These summary statistics prove invaluable in practice. For example, when comparing investment options, the expected value tells us the average return, while the variance indicates the risk. When designing systems, knowing the expected load and its variability helps us provision appropriate capacity.

Example: Variance of the Exponential Distribution

For our exponential distribution with PDF $f(x) = \lambda e^{-\lambda x}$, we'll compute the variance using the formula $\text{Var}(X) = E[X^2] - (E[X])^2$.

We already found that $E[X] = \frac{1}{\lambda}$, so $(E[X])^2 = \frac{1}{\lambda^2}$.

Now we need to find $E[X^2]$:

$$E[X^2] = \int_0^{\infty} x^2 \cdot \lambda e^{-\lambda x} \, dx = \lambda \int_0^{\infty} x^2 e^{-\lambda x} \, dx$$

We'll use integration by parts twice to evaluate $\int_0^{\infty} x^2 e^{-\lambda x} \, dx$.

First integration by parts: Let $u = x^2$ and $dv = e^{-\lambda x} \, dx$, then $du = 2x \, dx$ and $v = -\frac{1}{\lambda} e^{-\lambda x}$.

$$\begin{aligned} \int_0^{\infty} x^2 e^{-\lambda x} \, dx &= \left[-\frac{x^2}{\lambda} e^{-\lambda x}\right]_0^{\infty} + \frac{2}{\lambda} \int_0^{\infty} x e^{-\lambda x} \, dx \\[1em] &= 0 + \frac{2}{\lambda} \int_0^{\infty} x e^{-\lambda x} \, dx \end{aligned}$$

Note: The boundary term vanishes because $\lim_{x \to \infty} x^2 e^{-\lambda x} = 0$ (exponential decay dominates polynomial growth) and the term at $x=0$ is also zero.

Second integration by parts: For $\int_0^{\infty} x e^{-\lambda x} \, dx$, let $u = x$ and $dv = e^{-\lambda x} \, dx$, then $du = dx$ and $v = -\frac{1}{\lambda} e^{-\lambda x}$.

$$\begin{aligned} \int_0^{\infty} x e^{-\lambda x} \, dx &= \left[-\frac{x}{\lambda} e^{-\lambda x}\right]_0^{\infty} + \frac{1}{\lambda} \int_0^{\infty} e^{-\lambda x} \, dx \\[1em] &= 0 + \frac{1}{\lambda} \cdot \frac{1}{\lambda} \\[1em] &= \frac{1}{\lambda^2} \end{aligned}$$

Combining the results:

$$\begin{aligned} \int_0^{\infty} x^2 e^{-\lambda x} \, dx &= \frac{2}{\lambda} \cdot \frac{1}{\lambda^2} = \frac{2}{\lambda^3} \end{aligned}$$

Therefore:

$$\begin{aligned} E[X^2] &= \lambda \cdot \frac{2}{\lambda^3} \\[1em] &= \frac{2}{\lambda^2} \end{aligned}$$

Computing the variance:

$$\begin{aligned} \text{Var}(X) &= E[X^2] - (E[X])^2 \\[1em] &= \frac{2}{\lambda^2} - \frac{1}{\lambda^2} \\[1em] &= \frac{1}{\lambda^2} \end{aligned}$$

For $\lambda = 0.5$, this gives $\text{Var}(X) = \frac{1}{0.25} = 4$, and the standard deviation is $\sigma = \sqrt{4} = 2$.

Notice that for the exponential distribution, the standard deviation equals the mean ($\sigma = E[X] = \frac{1}{\lambda}$). This is a unique property of the exponential distribution and reflects its memoryless nature.the variability is proportional to the expected waiting time.

Example: Visualizing Expected Value and Variance

This example illustrates how expected value and variance characterize different aspects of a distribution. Two distributions can have the same mean but vastly different spreads.