Machine Learning in Trading Strategy Design › Unsupervised Learning for Market Structure
Quantitative FinanceAll chapters Machine Learning in Trading Strategy Design › Unsupervised Learning for Market StructureOne section · Sections in this chapter8 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Not all valuable patterns require labeled data. Unsupervised learning techniques discover hidden structure in market data without being told what to look for. These methods excel at grouping similar assets, identifying regime changes, and detecting anomalies that signal risk or opportunity. The absence of labels forces these algorithms to find patterns based on the intrinsic structure of the data, often revealing relationships that wouldn't be apparent from traditional analysis.
Clustering for Portfolio Construction
Hierarchical clustering and k-means provide data-driven alternatives to traditional sector classifications. Standard industry categories, like GICS sectors or NAICS codes, group companies based on what they do. But for portfolio construction, what matters is how assets behave together in the market, their return correlations and factor exposures. Two tech companies in the same GICS sector might have very different risk profiles if one is a growth stock sensitive to interest rates while the other is a mature value play.
Assets grouped by return covariance patterns may reveal economic relationships that standard industry categories miss. A cloud computing company might cluster more closely with commercial real estate (both sensitive to business investment cycles) than with social media companies (more dependent on advertising spending). By letting the data determine groupings, clustering can identify these unexpected but economically meaningful relationships:
import numpy as np
import pandas as pd
# Generate synthetic multi-asset returns with cluster structure
np.random.seed(42)
n_assets = 30
n_days = 252
# Create factor returns
tech_factor = np.random.normal(0.0005, 0.015, n_days)
value_factor = np.random.normal(0.0003, 0.012, n_days)
defensive_factor = np.random.normal(0.0002, 0.008, n_days)
market_factor = np.random.normal(0.0003, 0.01, n_days)
# Assign assets to clusters with varying factor exposures
asset_returns = np.zeros((n_days, n_assets))
cluster_assignments_true = []
for i in range(n_assets):
if i < 10: # Tech-like assets
asset_returns[:, i] = (
0.8 * tech_factor
+ 0.5 * market_factor
+ np.random.normal(0, 0.01, n_days)
)
cluster_assignments_true.append(0)
elif i < 20: # Value-like assets
asset_returns[:, i] = (
0.7 * value_factor
+ 0.6 * market_factor
+ np.random.normal(0, 0.012, n_days)
)
cluster_assignments_true.append(1)
else: # Defensive assets
asset_returns[:, i] = (
0.9 * defensive_factor
+ 0.3 * market_factor
+ np.random.normal(0, 0.006, n_days)
)
cluster_assignments_true.append(2)
returns_df = pd.DataFrame(
asset_returns, columns=[f"Asset_{i}" for i in range(n_assets)]
)
# Calculate correlation matrix
corr_matrix = returns_df.corr()from scipy.cluster.hierarchy import linkage
from scipy.spatial.distance import squareform
from sklearn.cluster import KMeans
# Perform hierarchical clustering on correlation distance
# Convert correlation to distance: d = sqrt(0.5 * (1 - corr))
distance_matrix = np.sqrt(0.5 * (1 - corr_matrix))
# Hierarchical clustering
linkage_matrix = linkage(squareform(distance_matrix), method="ward")
# K-means on return features (using covariance-based features)
cov_features = corr_matrix.values # Use correlation profile as features
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
cluster_labels = kmeans.fit_predict(cov_features)
from sklearn.metrics import adjusted_rand_score
ari = adjusted_rand_score(cluster_assignments_true, cluster_labels)Clustering Quality: Adjusted Rand Index: 1.000 (1.0 = perfect match with true clusters) Cluster Composition: Cluster 0: 10 assets - ['A10', 'A11', 'A12', 'A13', 'A14']... Cluster 1: 10 assets - ['A20', 'A21', 'A22', 'A23', 'A24']... Cluster 2: 10 assets - ['A0', 'A1', 'A2', 'A3', 'A4']...
The clustering identifies natural groupings in the data that can inform portfolio construction. The dendrogram on the left shows the hierarchical structure of asset relationships: assets that merge at lower heights are more similar, while those that only join at the top of the tree are quite different. The correlation heatmap on the right, reordered by cluster membership, reveals the block structure that clustering uncovers. Within each block (cluster), correlations are high, indicating that these assets move together. Between blocks, correlations are lower, creating diversification potential across clusters.
Rather than using arbitrary sector classifications that may not reflect current market dynamics, you can build diversified portfolios by selecting assets from different clusters. This approach diversifies risk exposures according to observed asset behavior rather than sector labels. When correlations change (as they often do during market stress), the clusters can be recomputed to maintain effective diversification.
Anomaly Detection for Risk Management
Unsupervised anomaly detection identifies unusual market behavior that might signal emerging risks or trading opportunities. The logic is that normal market conditions, while noisy, exhibit statistical regularities. When these regularities break down, when volatility spikes unexpectedly, when correlations shift dramatically, when the distribution of returns changes character, something unusual is happening. Detecting these anomalies early can help traders manage risk or capitalize on dislocations.
Isolation forests and autoencoders are particularly effective for high-dimensional financial data. Isolation forests work by recursively partitioning data; points that can be isolated quickly (with few partitions) are anomalous because they're far from the dense regions where most data lies. This approach is computationally efficient and doesn't require assumptions about the distribution of normal data:
from sklearn.ensemble import IsolationForest
def detect_market_anomalies(returns_df, contamination=0.05):
"""
Detect anomalous market conditions using Isolation Forest.
Parameters:
- returns_df: DataFrame of asset returns
- contamination: Expected proportion of anomalies
Returns:
- anomaly_scores: Lower = more anomalous
- anomaly_labels: -1 for anomalies, 1 for normal
"""
# Create cross-sectional features for each day
daily_features = pd.DataFrame(index=returns_df.index)
daily_features["cross_sectional_mean"] = returns_df.mean(axis=1)
daily_features["cross_sectional_std"] = returns_df.std(axis=1)
daily_features["cross_sectional_skew"] = returns_df.skew(axis=1)
daily_features["pct_positive"] = (returns_df > 0).mean(axis=1)
daily_features["dispersion"] = returns_df.max(axis=1) - returns_df.min(
axis=1
)
# Isolation Forest
iso_forest = IsolationForest(contamination=contamination, random_state=42)
anomaly_labels = iso_forest.fit_predict(daily_features)
anomaly_scores = iso_forest.decision_function(daily_features)
return daily_features, anomaly_scores, anomaly_labels
daily_features, anomaly_scores, anomaly_labels = detect_market_anomalies(
returns_df
)
The visualization highlights how the model isolates days with extreme cross-sectional properties. Normal market behavior (blue points) clusters in a characteristic region of the volatility-dispersion space. Anomalous days (red points) appear in unusual locations: perhaps extremely high volatility with moderate dispersion (a correlated sell-off), or moderate volatility with extreme dispersion (a sector rotation with some stocks surging while others crash). Each anomaly pattern tells a different story about what's happening in the market.
Anomaly detection serves multiple purposes in trading. It can identify potential market dislocations that create trading opportunities, since unusual conditions often precede mean-reversion. It can flag unusual portfolio behavior that warrants risk review, alerting managers when their positions are experiencing statistically unusual moves. And it can detect data quality issues before they corrupt model inputs, catching feed errors or stale prices that might otherwise trigger erroneous trades.