Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Machine Learning Techniques for Trading › Model Evaluation Metrics

Michael BrenndoerferJanuary 2, 20267 min read

Quantitative FinanceAll chapters

Part 13 of 15Machine Learning and Alternative Data

Chapter 69Machine Learning Techniques for Trading › Model Evaluation Metrics

Reading section 6 of 10 · Model Evaluation Metrics · 7 minSection 6 of 10 · 7 min

Section 6 of 10 · 7 min read

Machine Learning Techniques for Trading › Model Evaluation MetricsOne section · Sections in this chapter6 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Choosing appropriate evaluation metrics is critical for assessing whether a model will be useful in practice. The right metric depends on the prediction task and how the model will be used in trading decisions.

Regression MetricsLink Copied

For continuous predictions like return forecasts, common metrics include:

Mean Squared Error (MSE) measures the average squared difference between predictions and actual values:

\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2

where:

$n$ : total number of samples
$y_i$ : actual value for observation $i$
$\hat{y}_i$ : predicted value for observation $i$

MSE penalizes large errors heavily due to squaring, which may be appropriate when large prediction errors are particularly costly. In trading contexts, a model that occasionally makes huge errors might be more dangerous than one that makes consistent small errors, making MSE a sensible choice when tail risk matters.

Mean Absolute Error (MAE) is less sensitive to outliers:

\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|

where:

$n$ : total number of samples
$y_i$ : actual value
$\hat{y}_i$ : predicted value

Because MAE treats all errors linearly regardless of magnitude, it provides a more interpretable measure: the average MAE tells you the typical size of your prediction errors in the same units as your target variable.

R-squared ( $R^2$ ) measures the proportion of variance explained by the model:

R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}

where:

$\sum (y_i - \hat{y}_i)^2$ : residual sum of squares (unexplained variance)
$\sum (y_i - \bar{y})^2$ : total sum of squares (total variance)
$y_i$ : actual value for sample $i$
$\hat{y}_i$ : predicted value for sample $i$
$\bar{y}$ : mean of the actual values
$n$ : number of samples

This metric compares the model's errors to the variance of the target variable. An $R^2$ of 1 means the model perfectly predicts every observation; an $R^2$ of 0 means the model is no better than simply predicting the mean. Negative values indicate the model performs worse than this naive baseline.

In financial return prediction, $R^2$ values are typically very low (often below 0.05) because returns are inherently noisy. A seemingly tiny $R^2$ of 0.01 might still be valuable if its predictive power persists out of sample.

Classification MetricsLink Copied

For binary classification (predicting return direction, default events, etc.), metrics go beyond simple accuracy:

Accuracy measures the proportion of correct predictions:

\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}

where:

$TP$ : true positives (correctly predicted positive cases)
$TN$ : true negatives (correctly predicted negative cases)
$FP$ : false positives (incorrectly predicted positive cases)
$FN$ : false negatives (incorrectly predicted negative cases)

However, accuracy can be misleading with imbalanced classes. If positive returns occur 55% of the time, always predicting "positive" achieves 55% accuracy without any skill.

Precision and Recall focus on positive class predictions:

\text{Precision} = \frac{TP}{TP + FP}

where:

$TP$ : true positives
$FP$ : false positives

\text{Recall} = \frac{TP}{TP + FN}

where:

$TP$ : true positives
$FN$ : false negatives

Precision answers "of the times we predicted positive, how often were we right?" Recall answers "of all actual positives, how many did we catch?" These metrics often trade off against each other: being more selective (higher threshold) improves precision but reduces recall, and vice versa.

F1 Score balances precision and recall:

F1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}

where:

$\text{Precision}$ : proportion of positive predictions that were correct
$\text{Recall}$ : proportion of actual positives that were correctly identified

The F1 score is the harmonic mean of precision and recall, which penalizes extreme imbalances between the two. A model with perfect precision but terrible recall (or vice versa) will have a mediocre F1 score, while balanced performance yields higher scores.

ROC-AUC (Area Under the Receiver Operating Characteristic Curve) measures discrimination ability across all classification thresholds, and gives a threshold-independent assessment of model quality.

In[32]:

Code

from sklearn.metrics import (
    accuracy_score,
    classification_report,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)

# Train a model for evaluation
rf_eval = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42)
rf_eval.fit(X_train, y_train)

# Make predictions
y_pred = rf_eval.predict(X_test)
y_prob = rf_eval.predict_proba(X_test)[:, 1]

# Calculate metrics
metrics = {
    "Accuracy": accuracy_score(y_test, y_pred),
    "Precision": precision_score(y_test, y_pred),
    "Recall": recall_score(y_test, y_pred),
    "F1 Score": f1_score(y_test, y_pred),
    "ROC-AUC": roc_auc_score(y_test, y_prob),
}

# Generate report (computation)
class_report = classification_report(
    y_test, y_pred, target_names=["Down", "Up"]
)

Out[33]:

Console

Classification Performance Metrics:
-----------------------------------
Accuracy       : 0.5227
Precision      : 0.4912
Recall         : 0.6829
F1 Score       : 0.5714
ROC-AUC        : 0.4904

Detailed Classification Report:
              precision    recall  f1-score   support

        Down       0.58      0.38      0.46        47
          Up       0.49      0.68      0.57        41

    accuracy                           0.52        88
   macro avg       0.54      0.53      0.52        88
weighted avg       0.54      0.52      0.51        88

Accuracy alone is not enough to establish an edge, especially when the classes are uneven. Here the holdout ROC-AUC is about 0.49, which means the model does not reliably rank positive cases above negative ones in this split. That result calls for validation across several time-ordered folds rather than a claim of predictive signal from one test period.

Out[34]:

Visualization

ROC curve with false positive rate on the horizontal axis and true positive rate on the vertical axis. The model curve crosses around the dashed random-ranking diagonal and reports an AUC near 0.49. A marked point identifies the maximum Youden J statistic. — Receiver operating characteristic curve for the return direction predictor. The holdout AUC is about 0.49, close to and slightly below the random-ranking baseline; the marked point maximizes Youden's J statistic for this sample but does not establish a reliable trading edge.

Financial-Specific MetricsLink Copied

Standard ML metrics don't directly measure what matters most in trading: profitability. Consider supplementing with finance-specific measures:

Directional Accuracy Improvement compares the model's directional accuracy to a naive baseline (random guessing or always predicting the majority class).

Profit Factor measures the ratio of gross profits to gross losses when using the model's predictions for trading decisions.

Information Coefficient (IC) measures the correlation between predicted and actual returns, which directly relates to potential alpha generation.

In[35]:

Code

# Calculate financial metrics
# Simulated strategy: go long when model predicts up, short when predicts down
actual_returns = y_test_reg  # Actual returns in test period
predicted_direction = y_pred * 2 - 1  # Convert 0/1 to -1/+1

# Strategy returns
strategy_returns = actual_returns * predicted_direction

# Calculate metrics
cumulative_return = np.sum(strategy_returns)
sharpe_approximation = (
    np.mean(strategy_returns) / (np.std(strategy_returns) + 1e-8) * np.sqrt(252)
)
hit_rate = np.mean(strategy_returns > 0)
profit_factor = np.sum(strategy_returns[strategy_returns > 0]) / (
    abs(np.sum(strategy_returns[strategy_returns < 0])) + 1e-8
)

Out[36]:

Console

Financial Performance Metrics (Test Period):
---------------------------------------------
Cumulative Return:     0.0694
Annualized Sharpe:     0.57
Hit Rate:              52.27%
Profit Factor:         1.09

A profit factor above 1.0 indicates a profitable strategy, though practical trading would also account for transaction costs. The Sharpe ratio provides a risk-adjusted view; an annualized value around 1.0 or higher is typically targeted by quantitative funds.