Probability Basics › Independence and Conditional Probability

Real-world phenomena rarely occur in isolation. Events influence each other in complex ways. Probability theory provides tools to reason about relationships between events, distinguishing between cases where events are unrelated (independent) and cases where knowledge of one event changes our beliefs about another (dependent).

Two events $A$ and $B$ are independent if knowing whether one occurred provides no information about whether the other occurred. Mathematically, independence means that the probability of both events occurring equals the product of their individual probabilities:

This reads as: "the probability that both $A$ and $B$ occur equals the probability of $A$ multiplied by the probability of $B$."

Independence is a simplifying assumption that appears throughout statistical modeling. When we assume observations are independent, we can multiply their probabilities to compute joint likelihoods, enabling tractable inference in complex models. However, incorrectly assuming independence when dependence exists can lead to underestimating uncertainty and making overconfident predictions.

When events are dependent, we use conditional probability to express how the probability of one event changes given knowledge about another. The conditional probability of event $A$ given that event $B$ has occurred is written $P(A|B)$ and defined as:

This formula says that the conditional probability equals the probability of both events occurring divided by the probability of the conditioning event. Rearranging this definition gives us the multiplication rule:

Conditional probability forms the basis for Bayes' theorem, one of the most important results in probability theory and the foundation of Bayesian statistical inference:

Bayes' theorem allows us to "invert" conditional probabilities, computing $P(A|B)$ from $P(B|A)$. This inversion has many applications: in medical diagnosis, we might know the probability of a positive test given a disease, but we really want the probability of the disease given a positive test. In spam filtering, we observe words in an email and want to compute the probability the email is spam. In machine learning, we observe data and want to infer the probability of different model parameters.

Example: Conditional Probability and Bayes' Theorem

This example demonstrates how conditional probability updates our beliefs based on new information. We simulate a medical testing scenario where Bayes' theorem helps us understand the difference between the probability of a positive test given disease presence and the probability of disease given a positive test.

Let's work through a concrete example step by step. Suppose we have:

• Prior probability of disease: $P(D) = 0.01$ (1% of the population has the disease)
• Test sensitivity (true positive rate): $P(+|D) = 0.95$ (95% of sick people test positive)
• Test specificity (true negative rate): $P(-|\neg D) = 0.90$ (90% of healthy people test negative)

From specificity, we can derive the false positive rate:

Question: If someone tests positive, what is the probability they have the disease?

We want to find $P(D|+)$ using Bayes' theorem:

Step 1: Calculate $P(+)$ using the law of total probability

The total probability of testing positive considers both paths: having the disease and testing positive, or not having the disease and testing positive (false positive).

Step 2: Apply Bayes' theorem

Interpretation: Even with a positive test result, the probability of having the disease is only about 8.8%! This counterintuitive result occurs because:

1. The disease is rare (low prior probability)
2. The false positive rate (10%) applied to the large healthy population generates many false positives
3. These false positives "dilute" the true positives

This phenomenon is called the base rate fallacy and highlights why understanding conditional probability is essential for interpreting medical tests, machine learning classifiers, and any diagnostic system.

Despite the test appearing highly accurate (95% sensitivity, 90% specificity), the posterior probability of having the disease given a positive test is only about 8.8%. This counterintuitive result occurs because the disease is rare (1% prevalence). With 99 healthy people for every 1 sick person, even a 10% false positive rate generates many more false positives than true positives. This example illustrates why screening tests for rare conditions often require confirmation with follow-up testing.