Backtesting & Simulation: Frameworks for Strategy Validation › Building a Complete Backtest
Quantitative FinanceAll chapters Backtesting & Simulation: Frameworks for Strategy Validation › Building a Complete BacktestOne section · Sections in this chapter6 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's bring together all the concepts into a complete, realistic backtest implementation. We'll test a simple mean-reversion strategy while carefully avoiding biases and applying proper validation. This example demonstrates how the individual components combine into a coherent backtesting workflow.
Strategy Definition
Our strategy will trade based on Bollinger Bands, a mean-reversion approach where we buy when price falls below the lower band and sell when it rises above the upper band. The economic intuition is that prices tend to revert to their recent average, so extreme deviations represent temporary mispricings that will correct. When the price falls significantly below its moving average, the asset is potentially oversold and likely to bounce back. When the price rises significantly above its moving average, the asset is potentially overbought and likely to pull back.
class BollingerBandStrategy:
"""
Mean-reversion strategy using Bollinger Bands.
Buy signal: price closes below lower band (oversold)
Sell signal: price closes above upper band (overbought)
"""
def __init__(self, lookback=20, num_std=2):
self.lookback = lookback
self.num_std = num_std
def generate_signals(self, prices):
"""Generate trading signals from price data."""
# Calculate bands using only past data
rolling_mean = prices.rolling(window=self.lookback).mean()
rolling_std = prices.rolling(window=self.lookback).std()
upper_band = rolling_mean + self.num_std * rolling_std
lower_band = rolling_mean - self.num_std * rolling_std
# Shift to avoid look-ahead bias: signal based on yesterday's bands
upper_band = upper_band.shift(1)
lower_band = lower_band.shift(1)
rolling_mean = rolling_mean.shift(1)
# Generate signals
# +1: buy (price below lower band)
# -1: sell (price above upper band)
# 0: hold (between bands)
signals = pd.Series(0, index=prices.index)
signals[prices < lower_band] = 1 # Buy signal
signals[prices > upper_band] = -1 # Sell signal
return signals, rolling_mean, upper_band, lower_bandBacktest Engine with Transaction Costs
We'll incorporate transaction costs to make the simulation more realistic. Transaction costs are a critical component of any serious backtest because they can dramatically affect net profitability, especially for strategies that trade frequently. A strategy that looks highly profitable before costs may become marginal or even unprofitable after accounting for the friction of trading.
class BacktestEngine:
"""
Event-driven backtest engine with transaction cost modeling.
"""
def __init__(self, initial_capital=100000, transaction_cost=0.001):
"""
Parameters:
-----------
initial_capital : float
Starting capital in dollars
transaction_cost : float
Transaction cost as fraction of trade value (0.001 = 0.1% = 10bps)
"""
self.initial_capital = initial_capital
self.transaction_cost = transaction_cost
def run(self, prices, signals):
"""
Execute backtest given prices and signals.
Returns portfolio values and detailed trade log.
"""
n = len(prices)
# Portfolio state
cash = self.initial_capital
shares = 0
# Tracking arrays
portfolio_values = np.zeros(n)
positions = np.zeros(n)
trade_log = []
for t in range(n):
current_price = prices.iloc[t]
current_signal = signals.iloc[t]
# Execute trades based on signal change
if t > 0:
prev_signal = signals.iloc[t - 1]
# Enter long position
if current_signal == 1 and shares == 0:
# Calculate shares to buy (use all cash minus transaction cost)
trade_value = cash / (1 + self.transaction_cost)
shares = trade_value / current_price
cost = trade_value * self.transaction_cost
cash = cash - trade_value - cost
trade_log.append(
{
"date": prices.index[t],
"action": "BUY",
"shares": shares,
"price": current_price,
"cost": cost,
}
)
# Exit long position
elif current_signal == -1 and shares > 0:
trade_value = shares * current_price
cost = trade_value * self.transaction_cost
cash = cash + trade_value - cost
trade_log.append(
{
"date": prices.index[t],
"action": "SELL",
"shares": shares,
"price": current_price,
"cost": cost,
}
)
shares = 0
# Record state
portfolio_values[t] = cash + shares * current_price
positions[t] = shares
return pd.Series(portfolio_values, index=prices.index), trade_log
# Run the complete backtest
strategy = BollingerBandStrategy(lookback=20, num_std=2)
signals, ma, upper, lower = strategy.generate_signals(prices_df["Stock_A"])
engine = BacktestEngine(initial_capital=100000, transaction_cost=0.001)
portfolio_values, trades = engine.run(prices_df["Stock_A"], signals)
total_costs = sum(t["cost"] for t in trades)Backtest completed with 18 trades Sample trades: 2020-02-21: BUY 1149.53 shares @ $86.91 2020-04-06: SELL 1149.53 shares @ $85.86 2020-05-13: BUY 1182.46 shares @ $83.30 2020-06-17: SELL 1182.46 shares @ $87.43 2020-07-24: BUY 1280.39 shares @ $80.58 Total transaction costs paid: $2289.18
The trade log allows us to verify that the strategy executes as intended, buying and selling at the correct signals. Each trade record includes the date, action type, number of shares, execution price, and transaction cost. This detailed logging is essential for debugging and for understanding how the strategy behaves in different market conditions. The total transaction costs figure highlights the friction that will reduce net returns, a critical factor often overlooked in theoretical models that assume frictionless markets.
Comparing In-Sample vs Out-of-Sample Performance
Finally, let's validate our strategy using proper train-test methodology. We'll optimize the strategy parameters on training data and then evaluate performance on held-out test data to assess how much of the apparent alpha persists out of sample rather than reflects overfitting.
def compare_in_out_sample(
prices,
strategy_class,
train_fraction=0.7,
param_grid={"lookback": [10, 15, 20, 25, 30], "num_std": [1.5, 2.0, 2.5]},
):
"""
Compare in-sample (optimized) vs out-of-sample performance.
"""
# Split data
split_idx = int(len(prices) * train_fraction)
train_prices = prices.iloc[:split_idx]
test_prices = prices.iloc[split_idx:]
# Find best parameters on training data
best_params = None
best_in_sample_sharpe = -np.inf
engine = BacktestEngine(transaction_cost=0.001)
for lookback in param_grid["lookback"]:
for num_std in param_grid["num_std"]:
strategy = strategy_class(lookback=lookback, num_std=num_std)
signals, _, _, _ = strategy.generate_signals(train_prices)
pv, _ = engine.run(train_prices, signals)
returns = pv.pct_change().dropna()
if len(returns) > 0:
sharpe = calculate_sharpe_ratio(returns)
if sharpe > best_in_sample_sharpe:
best_in_sample_sharpe = sharpe
best_params = {"lookback": lookback, "num_std": num_std}
# Evaluate best parameters on test data
best_strategy = strategy_class(**best_params)
# Re-run on training data with best params
train_signals, _, _, _ = best_strategy.generate_signals(train_prices)
train_pv, _ = engine.run(train_prices, train_signals)
train_returns = train_pv.pct_change().dropna()
# Run on test data
test_signals, _, _, _ = best_strategy.generate_signals(test_prices)
test_pv, _ = engine.run(test_prices, test_signals)
test_returns = test_pv.pct_change().dropna()
return {
"best_params": best_params,
"in_sample_sharpe": calculate_sharpe_ratio(train_returns),
"out_of_sample_sharpe": calculate_sharpe_ratio(test_returns),
"in_sample_return": (train_pv.iloc[-1] / train_pv.iloc[0] - 1) * 100,
"out_of_sample_return": (test_pv.iloc[-1] / test_pv.iloc[0] - 1) * 100,
"train_pv": train_pv,
"test_pv": test_pv,
}
validation_results = compare_in_out_sample(
prices_df["Stock_A"], BollingerBandStrategy
)In-Sample vs Out-of-Sample Validation
==================================================
Best parameters found: {'lookback': 30, 'num_std': 2.0}
In-Sample Performance (training period):
Sharpe Ratio: 1.29
Total Return: 35.09%
Out-of-Sample Performance (test period):
Sharpe Ratio: 0.32
Total Return: 3.45%
Performance degradation:
Sharpe drop: 0.97
The gap between in-sample and out-of-sample performance is typical and expected. This performance degradation, where out-of-sample results are worse than in-sample results, occurs because some of the patterns captured during optimization are noise rather than signal. A strategy that shows similar performance in both periods is more likely to generalize than one where in-sample results far exceed out-of-sample results. A large gap suggests that a significant portion of the apparent alpha reflects overfitting to the training data rather than market inefficiencies that will persist going forward.