Hierarchical Clustering › Implementation in Scikit-learn
Machine Learning from ScratchAll chapters Hierarchical Clustering › Implementation in Scikit-learnOne section · Sections in this chapter5 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Scikit-learn implements hierarchical clustering through the AgglomerativeClustering class. The class is optimized for performance and follows the same estimator interface as other scikit-learn models, so it can be used in machine learning pipelines.
Let's walk through a complete example that demonstrates how to use hierarchical clustering with different linkage criteria and evaluate the results.
Step 1: Import Required Libraries
First, we import the necessary libraries for our hierarchical clustering implementation:
Step 2: Generate and Prepare Sample Data
We'll create a synthetic dataset with known cluster structure to evaluate our clustering performance:
# Generate sample data with 4 distinct clusters
np.random.seed(42)
X, y_true = make_blobs(
n_samples=300, centers=4, cluster_std=1.5, random_state=42
)
# Standardize the data - crucial for clustering algorithms
X = StandardScaler().fit_transform(X)Dataset shape: (300, 2) Number of true clusters: 4
The dataset contains 300 data points with 2 features, organized into 4 true clusters. The shape (300, 2) indicates we have 300 samples, each with 2 features, which is well-suited for visualization and understanding how hierarchical clustering works. Standardization ensures that both features contribute equally to the distance calculations, preventing features with larger scales from dominating the clustering process.
Step 3: Apply Hierarchical Clustering with Different Linkage Criteria
Now we'll compare different linkage criteria to see how they affect clustering results:
from sklearn.cluster import AgglomerativeClustering
# Define linkage methods to compare
linkage_methods = ["single", "complete", "average", "ward"]
clustering_results = {}
# Apply hierarchical clustering with each linkage method
for method in linkage_methods:
# Note: ward linkage only works with euclidean distance
clustering = AgglomerativeClustering(
n_clusters=4, # Number of clusters to find
linkage=method, # Linkage criterion
metric="euclidean", # Distance metric
)
# Fit the model and predict cluster labels
y_pred = clustering.fit_predict(X)
clustering_results[method] = y_predStep 4: Evaluate Clustering Performance
Let's calculate performance metrics for each linkage method to compare their effectiveness:
from sklearn.metrics import adjusted_rand_score, silhouette_score
# Calculate performance metrics for each method
metrics = {}
for method, y_pred in clustering_results.items():
ari = adjusted_rand_score(y_true, y_pred)
silhouette = silhouette_score(X, y_pred)
metrics[method] = {"ari": ari, "silhouette": silhouette}Single linkage: Adjusted Rand Index: 0.708 Silhouette Score: 0.558 Complete linkage: Adjusted Rand Index: 0.982 Silhouette Score: 0.698 Average linkage: Adjusted Rand Index: 0.982 Silhouette Score: 0.698 Ward linkage: Adjusted Rand Index: 0.982 Silhouette Score: 0.698
The results demonstrate how different linkage criteria perform on our dataset. The Adjusted Rand Index (ARI) measures how well the clustering matches the true labels, with values ranging from -1 to 1. An ARI of 1 indicates complete agreement with the true clusters, while values closer to 0 suggest random clustering. The Silhouette Score measures cluster quality based on how well-separated clusters are, with values ranging from -1 to 1. Values closer to 1 indicate well-separated, cohesive clusters, while values below 0.2 suggest clusters may not be well-separated.
In this example, complete linkage, average linkage, and Ward linkage all recover the true cluster structure well, with ARI scores above 0.9 and silhouette scores above 0.5. Single linkage performs worse on this type of data because one close connection can join two broader groups before all four natural clusters have been cleanly separated. The performance differences highlight the importance of choosing the appropriate linkage criterion based on your data characteristics and clustering goals.
Step 5: Visualize Clustering Results
Let's visualize the clustering results to see how different linkage criteria affect the cluster shapes:
You can clearly see how different linkage criteria create different results. Single linkage is pulled by local nearest-neighbor connections and produces an imbalanced solution here, while complete linkage, average linkage, and Ward linkage recover the four compact groups much more cleanly.
Key Parameters
Below are some of the main parameters that affect how the model works and performs.
-
n_clusters: The number of clusters to find (default: 2). This determines where to cut the dendrogram. Should be specified in advance, unlike the theoretical algorithm that creates a complete hierarchy. Use the dendrogram visualization to identify natural cut points by looking for significant gaps in merging distances. If uncertain, try multiple values and evaluate using metrics like silhouette score. -
linkage: The linkage criterion ('single', 'complete', 'average', or 'ward', default: 'ward'). Single linkage tends to create elongated, chain-like clusters and is sensitive to noise. Complete linkage creates compact, spherical clusters but may break up natural elongated clusters. Average linkage provides a balanced approach and is often a good default choice. Ward linkage minimizes within-cluster variance and works well for spherical clusters of similar sizes, but only works with Euclidean distance. -
metric: The distance metric to use (default: 'euclidean'). Common options include 'euclidean', 'manhattan', 'cosine', and 'l1'. Note that Ward linkage only works with Euclidean distance. Choose based on your data characteristics: Euclidean for continuous features, Manhattan for features with outliers, and cosine for high-dimensional sparse data. -
compute_full_tree: Whether to compute the full tree (default: 'auto'). Set to 'auto' to compute the full tree when n_clusters is small relative to n_samples, or True to compute the full tree for dendrogram visualization. Setting to False can save computation time when you only need a specific number of clusters. -
distance_threshold: Distance threshold above which clusters will not be merged (default: None). If set, n_clusters should be None. This allows you to cut the dendrogram at a specific distance rather than specifying the number of clusters.
Key Methods
The following are the most commonly used methods for interacting with the model.
-
fit(X): Fits the hierarchical clustering model to the data X and builds the cluster hierarchy. This method computes the distance matrix and performs the agglomerative clustering process. -
fit_predict(X): Fits the model to the data X and returns cluster labels for each data point in a single step. This is the most commonly used method as it combines fitting and prediction in one operation. -
predict(X): Predicts cluster labels for new data points using the fitted model. Note that hierarchical clustering doesn't naturally support prediction on new data, so this method may not work as expected. For new data points, consider using the fitted model to guide parameter selection for methods like K-means that support prediction. -
get_params(): Returns the parameters of the clustering model for inspection and debugging. Useful for understanding the current configuration of the model.
Alternative Implementation
For cases where you need more control over the clustering process or want to work with the full dendrogram, you can use scipy's hierarchical clustering functions directly. This approach gives you access to the complete linkage matrix and allows you to extract clusters at any level of the hierarchy:
from scipy.cluster.hierarchy import fcluster, linkage
# Generate the linkage matrix using average linkage
linkage_matrix = linkage(X, method="average")
# Extract clusters at different levels by specifying the number of clusters
clusters_2 = fcluster(linkage_matrix, 2, criterion="maxclust")
clusters_4 = fcluster(linkage_matrix, 4, criterion="maxclust")
# Count points in each cluster for both clusterings
cluster_counts_2 = np.unique(clusters_2, return_counts=True)
cluster_counts_4 = np.unique(clusters_4, return_counts=True)Clusters with 2 groups: Cluster labels: [1 2] Point counts: [ 75 225] Clusters with 4 groups: Cluster labels: [1 2 3 4] Point counts: [75 75 75 75]
The output shows how the same linkage matrix can be used to extract different numbers of clusters. With 2 clusters, the data is divided into two broad groups, while with 4 clusters, we get more granular groupings. The point counts show how many data points belong to each cluster, which helps assess whether clusters are balanced or if some clusters are much larger than others. This flexibility to explore different cluster numbers from a single linkage matrix is one of the key advantages of hierarchical clustering.
This alternative approach gives you direct access to the linkage matrix and dendrogram, allowing for more detailed analysis of the clustering structure. The fcluster function lets you extract clusters at any level of the hierarchy by specifying either the number of clusters or a distance threshold.