Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Statistical Data Analysis and Inference › Hypothesis Testing

Michael BrenndoerferOctober 22, 20258 min read

Quantitative FinanceAll chapters

Part 1 of 15Mathematical Foundations

Chapter 4Statistical Data Analysis and Inference › Hypothesis Testing

Reading section 4 of 9 · Hypothesis Testing · 8 minSection 4 of 9 · 8 min

Section 4 of 9 · 8 min read

Statistical Data Analysis and Inference › Hypothesis TestingOne section · Sections in this chapter4 / 9 ↓

Sections in this chapter9

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Hypothesis testing provides a framework for making decisions based on data. In finance, you might ask: "Is this strategy's return significantly different from zero?" or "Does this factor predict returns?" These questions require a systematic approach that accounts for uncertainty in sample-based estimates.

The FrameworkLink Copied

A hypothesis test has four components:

Null hypothesis ( $H_0$ ): The default assumption, typically "no effect" or "no difference"
Alternative hypothesis ( $H_1$ or $H_a$ ): What you're trying to demonstrate
Test statistic: A number computed from data that measures evidence against $H_0$
P-value: Under the specified null model, the probability that the prespecified test statistic is at least as extreme as the observed value

Hypothesis testing starts by assuming the null model and asking how unusual the observed test statistic is under a prespecified one- or two-sided extremeness rule. A sufficiently small tail probability leads us to reject the null; otherwise, we fail to reject it. Note the asymmetry: we never "accept" the null hypothesis; we merely fail to find sufficient evidence against it.

The p-value is not the probability of the observed dataset or the probability that the null hypothesis is true. It is a tail probability for the chosen statistic under the null model. A p-value of 0.01, for example, means that the null model assigns 1% probability to statistic values at least as extreme as the observed value according to the prespecified test.

If the p-value is below a threshold (commonly $\alpha = 0.05$ ), we reject $H_0$ in favor of $H_1$ . The choice of $\alpha = 0.05$ is a convention, not a law of nature. The significance level controls the Type I error rate under the null. Type II error also depends on the specified alternative, effect size, sample size, variability, and test, so choosing $\alpha$ alone does not determine a universal balance between the two errors.

Testing if Mean Return Differs from ZeroLink Copied

A common hypothesis test in strategy evaluation asks whether a modeled population mean return differs from zero under specified sampling assumptions. The test does not by itself establish persistent skill, a strategy edge, economic profitability, or tradability.

Under the null hypothesis that the true mean is zero:

H_0: \mu = 0 \quad \text{vs} \quad H_1: \mu \neq 0

The null hypothesis states that the modeled population mean return is zero. The alternative states that it is not zero (it could be positive or negative). This is a two-tailed test because we're interested in deviations from zero in either direction.

The test statistic is:

t = \frac{\bar{r} - 0}{s/\sqrt{n}} = \frac{\bar{r}}{\widehat{SE}(\bar{r})}

where:

$t$ : test statistic
$\bar{r}$ : sample mean return
$s$ : sample standard deviation (estimate of $\sigma$ )
$\widehat{SE}(\bar{r}) = s/\sqrt{n}$ : estimated standard error

The t-statistic requires $n\geq2$ and $s>0$ . It measures how many estimated standard errors the sample mean is from zero. A t-statistic of 2 means the observed mean is 2 estimated standard errors away from zero. Under a valid t model, the exact tail probability depends on the degrees of freedom: the two-sided probability is about 4.7% at 250 degrees of freedom, but about 29.5% at one degree of freedom. At fixed degrees of freedom and a valid null model, larger absolute statistics give smaller two-sided p-values.

Under $H_0$ , this statistic has an exact t-distribution with $n-1$ degrees of freedom when the observations are i.i.d. normal. With nonnormal i.i.d. data, the test is generally approximate for sufficiently large samples. Serial dependence or heteroskedasticity can invalidate the standard error $s/\sqrt n$ ; long-run-variance estimators or an appropriate bootstrap are then needed.

In[24]:

Code

# Test if mean return is significantly different from zero
# Variables mean_daily, standard_error, and n were computed in "The First Moment" and "The Standard Error" sections
t_statistic = mean_daily / standard_error
degrees_freedom = n - 1

# Two-tailed p-value
p_value = 2 * stats.t.sf(abs(t_statistic), degrees_freedom)

# Using scipy's ttest_1samp for verification
t_stat_scipy, p_value_scipy = stats.ttest_1samp(log_returns, 0)

Out[25]:

Console

Hypothesis Test: Is mean return different from zero?
--------------------------------------------------
Mean daily return: 0.000199
Standard error: 0.001219
t-statistic: 0.1629
Degrees of freedom: 250
P-value (two-tailed): 0.8707

Conclusion: Fail to reject H₀ at α=0.05. Insufficient evidence that mean differs from zero.

One-Tailed vs. Two-Tailed TestsLink Copied

The test above was two-tailed, asking whether the mean differs from zero in either direction. Often in finance, you care specifically about whether returns are positive:

H_0: \mu \leq 0 \quad \text{vs} \quad H_1: \mu > 0

This formulation changes the question from "is there an effect?" to "is there a positive effect?" The null hypothesis now includes all non-positive values, and we only reject it if we find strong evidence of positive returns.

For a symmetric test statistic, a one-tailed p-value is half the two-tailed p-value only when the direction was specified in advance and the observed statistic points in that direction. Here we compute the upper-tail probability directly:

In[26]:

Code

# One-tailed test: is mean return significantly positive?
p_value_one_tailed = stats.t.sf(t_statistic, degrees_freedom)

Out[27]:

Console

One-Tailed Test: Is mean return significantly positive?
--------------------------------------------------
t-statistic: 0.1629
P-value (one-tailed): 0.4354
Conclusion: Insufficient evidence that returns are significantly positive.

The Multiple Testing ProblemLink Copied

In practice, quantitative researchers test many strategies, factors, or parameters. If all 100 null hypotheses are true and each test rejects with probability exactly $\alpha = 0.05$ under its null, the expected number of false rejections is 5. If each test is merely valid at level 0.05, the expectation is at most 5. This is the multiple testing problem, and it represents one of the most significant challenges in quantitative finance research.

The mathematics is straightforward, but its implications are serious. With a 5% false positive rate per test, the probability of at least one false positive across $m$ independent tests is $1 - (1-0.05)^m$ . For 100 tests, this exceeds 99%. In other words, finding at least one "significant" result is virtually guaranteed even when no true effects exist.

Multiple Testing Corrections

Bonferroni correction: Divide $\alpha$ by the number of tests. Testing 100 strategies? Use $\alpha = 0.05/100 = 0.0005$ .

Benjamini-Hochberg: Under independent tests, or under certain positive-dependence conditions, the standard step-up procedure controls the false discovery rate (FDR) rather than the family-wise error rate.

The Bonferroni correction is simple but can be conservative; how conservative it is depends on the joint dependence structure of the tests. The Benjamini-Hochberg procedure offers a different approach: under independence or suitable positive dependence, it controls the expected false-discovery proportion at the chosen level (using zero when there are no discoveries), rather than the probability of any false discovery. Under arbitrary dependence, use an adjustment or procedure with a guarantee that matches the dependence structure.

In[28]:

Code

# Simulate multiple testing problem
np.random.seed(42)
n_strategies = 100
n_days = 252

# All strategies have zero true mean (null is true for all)
strategy_returns = np.random.normal(0, 0.02, (n_strategies, n_days))

# Test each strategy
p_values = []
for i in range(n_strategies):
    _, pval = stats.ttest_1samp(strategy_returns[i], 0)
    p_values.append(pval)

p_values = np.array(p_values)

Out[29]:

Console

Multiple Testing Simulation (all strategies have zero true mean)
------------------------------------------------------------
Strategies tested: 100
Strategies 'significant' at α=0.05: 4
Expected false positives: 5

After Bonferroni correction (α=0.0005):
Strategies 'significant': 0

Out[30]:

Visualization

Histogram of p-values from 100 tests under the null, with a vertical line at 0.05 and a shaded rejection region highlighting expected false positives. — Distribution of p-values when testing 100 strategies that all have zero true mean. Under the null hypothesis, p-values should be uniformly distributed between 0 and 1. The red shaded region shows the rejection area at α = 0.05, capturing approximately 5% of tests as false positives.

This simulation shows why backtesting results should be viewed skeptically. When you've tested many variations and found one that "works," it may simply be a false positive. Selecting the maximum from 100 null strategies can make the winner appear stronger than a typical null result because it was selected for favorable noise. This selection effect is one form of data dredging and requires careful attention.