Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Probability Basics › Probability Distributions

Michael BrenndoerferUpdated December 25, 202515 min read

Machine Learning from ScratchAll chapters

Part 1 of 9Statistics 101

Chapter 3Probability Basics › Probability Distributions

Reading section 4 of 12 · Probability Distributions · 15 minSection 4 of 12 · 15 min

Section 4 of 12 · 15 min read

Probability Basics › Probability DistributionsOne section · Sections in this chapter4 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A probability distribution describes how probabilities are spread across the possible values of a random variable. It tells us which outcomes are likely and which are rare. Different types of random phenomena follow different patterns, and recognizing these patterns helps us model real-world processes effectively.

For discrete random variables, we use a probability mass function (PMF) that assigns probabilities to each possible value. The PMF must satisfy two conditions: all probabilities must be non-negative, and they must sum to one across all possible outcomes. For continuous random variables, we use a probability density function (PDF) that describes the relative likelihood of values in different regions. Unlike discrete probabilities, the PDF itself does not give probabilities directly. Instead, we integrate the PDF over an interval to find the probability that the variable falls within that range.

Example: PMF vs PDF - Understanding Discrete and Continuous Distributions

This visualization contrasts how probabilities are represented for discrete versus continuous random variables. The discrete case shows exact probabilities for specific outcomes (the height of each bar), while the continuous case shows a density curve where probabilities are areas under the curve.

Out[2]:

Visualization

Six equal-height bars show that every fair-die outcome from one through six has probability 0.167, with a legend identifying the bars as P(X = k). — Probability Mass Function (PMF): A fair die roll where each outcome has probability 1/6. The bars show exact probabilities at discrete points. The heights sum to 1.0, and we can read probabilities directly from the bar heights. For example, P(X=3) = 0.167.

Uniform density from zero to six with a legend distinguishing the full distribution from the interval two to four, whose shaded area is labeled 0.333. — Probability Density Function (PDF): A continuous uniform distribution on [0,6]. The curve shows probability density, not probability itself. The area under the entire curve equals 1.0. To find probabilities, we calculate areas: for example, P(2 < X < 4) equals the shaded area, which is 0.333.

Several probability distributions appear repeatedly due to their theoretical properties and empirical relevance. The normal distribution (Gaussian distribution) describes many natural phenomena and plays a central role in statistical theory due to the Central Limit Theorem. The binomial distribution models the number of successes in a fixed number of independent trials, such as counting how many emails out of 100 are spam. The Poisson distribution describes the number of events occurring in a fixed interval of time or space, useful for modeling rare events like server failures or customer arrivals. The exponential distribution models waiting times between events in a Poisson process, such as time between website visits.

Choosing the distribution that best describes your data helps you build an appropriate statistical model. Misspecifying the distribution can lead to incorrect inferences, poor predictions, and unreliable uncertainty estimates.

PMF in Practice

The discrete PMF example above is somewhat artificial. We defined specific probabilities for each outcome that conveniently sum to 1. In practice, PMFs are typically derived from theoretical distributions, like binomial or Poisson, that have mathematical formulas governing the probabilities. We use this simple example to build intuition before introducing these more complex distributions.

Example: Comparing Probability Distributions

The following visualization shows three fundamental probability distributions: the normal distribution describing continuous measurements, the binomial distribution describing count data from repeated trials, and the Poisson distribution describing rare event counts.

Out[3]:

Visualization

Symmetric bell-shaped standard normal density centered at zero. — Normal Distribution (μ=0, σ=1): The bell-shaped curve describes many natural phenomena due to the Central Limit Theorem. This continuous distribution is symmetric around its mean, with approximately 68% of values within one standard deviation.

Discrete binomial probabilities for the number of successes in 20 fair trials, centered around 10. — Binomial Distribution (n=20, p=0.5): Models the number of successes in a fixed number of independent trials. Here we show the distribution of heads in 20 coin flips.

Right-skewed Poisson probabilities for event counts when the expected count is three. — Poisson Distribution (λ=3): Describes the number of rare events in a fixed interval. The distribution is skewed right for small λ values and becomes more symmetric as λ increases, useful for modeling arrivals, failures, or other count phenomena.

Calculating Probabilities from PMFsLink Copied

Once we have defined a probability distribution, we can calculate the probability of specific outcomes or ranges of outcomes. The approach differs depending on whether we're working with discrete or continuous distributions.

For discrete distributions, we use the Probability Mass Function (PMF), which gives us $P(X = x)$ , the probability that the random variable equals exactly a specific value. The PMF must satisfy two key properties:

$P(X = x) \geq 0$ for all values $x$ (probabilities are non-negative)
$\sum_{\text{all } x} P(X = x) = 1$ (probabilities sum to 1)

To find the probability of a range of outcomes, we simply sum the individual probabilities. For example, $P(2 \leq X \leq 5) = P(X=2) + P(X=3) + P(X=4) + P(X=5)$ .

For continuous distributions, we use the Probability Density Function (PDF) along with integration to find probabilities over intervals. Unlike the PMF, the PDF does not give probabilities directly. Instead, we must integrate over an interval:

P(a \leq X \leq b) = \int_a^b f(x) \, dx

Probability Density Function

Here, $f(x)$ denotes the probability density function (PDF) of the continuous random variable $X$ . But where does this function come from?

Each continuous distribution has its own specific PDF formula derived from the properties of that distribution. For example, the normal distribution has the familiar bell-curve formula $f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$ .

When using statistical software like scipy.stats, these PDF formulas are built-in. Calling stats.norm.pdf(x) evaluates this formula for you. The key point is that while a PDF tells us the density of probability at each point, we must integrate it over an interval to get actual probabilities.

As we recall, integration is a mathematical operation that calculates the area under a curve. Imagine drawing the PDF curve on a graph and shading the region between the curve and the x-axis from point $a$ to point $b$ . The integral $\int_a^b f(x) \, dx$ computes exactly this shaded area. In the context of probability, this area represents the probability that the random variable falls within that interval. Since the total area under the entire PDF curve equals 1 (representing 100% probability), the area under any portion of the curve tells us what fraction of the total probability lies in that range.

An important consequence is that for continuous distributions, $P(X = x) = 0$ for any specific value $x$ , where $x$ is any particular point in the continuous range of possible values. This may seem counterintuitive, but it makes sense when you consider that there are infinitely many possible values. What matters is the probability of falling within an interval, not at a specific point.

The Cumulative Distribution Function (CDF) provides another useful tool for calculating probabilities. The CDF, denoted $F(x)$ , gives the probability that the random variable is less than or equal to $x$ :

F(x) = P(X \leq x)

For continuous distributions, the CDF is obtained by integrating the PDF from $-\infty$ to $x$ :

F(x) = \int_{-\infty}^{x} f(t) \, dt

In other words, the CDF at any point equals the total area under the PDF curve to the left of that point.

Example: Calculating Probabilities with the Binomial Distribution

Let's work through a concrete example to make these concepts clear. Imagine we're flipping a fair coin 10 times and want to calculate various probabilities. This scenario follows a binomial distribution with parameters $n = 10$ (number of trials) and $p = 0.5$ (probability of success/heads on each flip).

The binomial PMF gives us the exact probability of observing exactly $k$ heads in 10 flips:

P(X = k) = \binom{10}{k} (0.5)^k (0.5)^{10-k} = \binom{10}{k} (0.5)^{10}

Where $\binom{10}{k} = \frac{10!}{k!(10-k)!}$ is the binomial coefficient (number of ways to get exactly $k$ heads in 10 flips).

Let's understand where this formula comes from:

$(0.5)^k$ : The probability of getting exactly $k$ heads, since each head has probability 0.5
$(0.5)^{10-k}$ : The probability of getting exactly $10-k$ tails for the remaining flips
$\binom{10}{k}$ : The number of different ways to arrange $k$ heads among 10 flips (since we don't care about the order, just the total count)

Multiplying these together gives us the total probability of observing exactly $k$ heads.

Binomial Distribution Formula

This formula is specific to the binomial distribution, which models the number of successes in a fixed number of independent trials (like coin flips). The general binomial PMF formula comes from probability theory literature and is given by $P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}$ , where $n$ is the number of trials, $k$ is the number of successes, and $p$ is the probability of success on each trial.

Let's calculate some specific probabilities and visualize the entire distribution:

Out[4]:

Visualization

Binomial probability bars for zero through ten heads, with five heads highlighted at probability 0.246 and a legend defining the bars as P(X = k). — Binomial PMF (n=10, p=0.5): The probability mass function shows exact probabilities for each possible number of heads from zero through ten. The distribution is symmetric around its expected value, and the most likely outcome is highlighted at P(X=5)≈0.246.

Increasing binomial cumulative probability step curve with dashed guides and labels at zero, two, five, seven, and ten heads, plus a legend defining F(k) as P(X ≤ k). — Binomial CDF (n=10, p=0.5): The cumulative distribution function shows the probability of getting at most k heads. Dashed guides and labels mark representative cumulative probabilities from F(0)=0.001 through F(10)=1.000, and the legend states F(k)=P(X≤k).

Now let's use this distribution to calculate specific probabilities. Here are some common types of probability questions we might ask:

Exact probabilities (using PMF):

$P(X = 0)$ = Probability of no heads (all tails)
$P(X = 5)$ = Probability of exactly 5 heads
$P(X = 10)$ = Probability of all heads

Cumulative probabilities (using CDF):

$P(X \leq 3)$ = Probability of 3 or fewer heads
$P(X \geq 7)$ = Probability of 7 or more heads
$P(X > 5)$ = Probability of more than 5 heads

Range probabilities (using CDF subtraction):

$P(3 \leq X \leq 7)$ = Probability of 4 through 7 heads
$P(2 < X < 8)$ = Probability of 3 through 7 heads

Let's calculate these probabilities numerically:

Out[5]:

Console

Exact Probabilities (PMF):
P(X = 0)  = 0.000977  # No heads
P(X = 5)  = 0.246094  # Exactly 5 heads
P(X = 10) = 0.000977 # All heads

Cumulative Probabilities (CDF):
P(X ≤ 3)  = 0.171875  # At most 3 heads
P(X ≥ 7)  = 0.171875  # At least 7 heads
P(X > 5)  = 0.376953  # More than 5 heads

Range Probabilities (CDF subtraction):
P(3 ≤ X ≤ 7) = 0.890625  # 3 to 7 heads
P(2 < X < 8)  = 0.890625  # 3 to 7 heads

Notice how the probabilities behave:

The most likely outcome is 5 heads (24.6% probability), which matches our intuition for a fair coin
Extreme outcomes (0 or 10 heads) are very unlikely (0.1% each)
The distribution is symmetric around the mean because p = 0.5
Cumulative probabilities increase steadily as we move from left to right
Range probabilities are calculated by subtracting CDF values: P(a ≤ X ≤ b) = F(b) - F(a-1)

This example demonstrates why probability calculations are so important: they allow us to quantify uncertainty and make informed decisions based on the likelihood of different outcomes.

Using the scipy.stats Package for Probability Calculations

In the calculations above, we used scipy.stats.binom.pmf() and scipy.stats.binom.cdf() to compute probabilities. These functions efficiently implement the formulas we discussed earlier.

If we were to calculate these from scratch:

PMF (Probability Mass Function): We'd compute $P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}$ using math.comb(n, k) for the binomial coefficient and standard arithmetic for the rest.
CDF (Cumulative Distribution Function): We'd sum up the PMF values: $P(X \leq k) = \sum_{i=0}^{k} P(X = i)$ , which involves computing and adding each individual probability.

The scipy.stats package provides optimized implementations that handle numerical precision and edge cases, making it the preferred choice for practical work. However, understanding the underlying formulas is essential for building intuition about how these probabilities behave.