K-means Clustering › Implementation
Machine Learning from ScratchAll chapters K-means Clustering › ImplementationOne section · Sections in this chapter7 / 15 ↓
This section provides a step-by-step tutorial for implementing K-means clustering using scikit-learn. We'll work through a complete example, from data preparation and model fitting to evaluation.
Step 1: Data Preparation and Setup
First, we'll import the necessary libraries and create a synthetic dataset with known cluster structure. This allows us to validate our implementation against ground truth.
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.preprocessing import StandardScaler
# Set random seed for reproducibility
np.random.seed(42)
# Generate synthetic data with 4 well-separated clusters
X, y_true = make_blobs(
n_samples=300, centers=4, cluster_std=1.5, random_state=42
)
# Standardize the data - crucial for K-means due to distance-based calculations
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)The dataset contains 300 samples with 2 features, organized into 4 natural clusters. Standardization ensures that both features contribute equally to the distance calculations, preventing features with larger scales from dominating the clustering process.
Dataset shape: (300, 2) Number of true clusters: 4
The output confirms our data setup is correct. The shape shows we have 300 data points with 2 features, and the 4 unique cluster labels match our synthetic data generation parameters. This provides a solid foundation for testing K-means clustering.
Step 2: Basic K-means Implementation
Now we'll implement K-means clustering with the known optimal number of clusters. The algorithm will iteratively assign points to clusters and update centroids until convergence.
# Initialize and fit K-means with 4 clusters
kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
kmeans.fit(X_scaled)
# Extract results
cluster_labels = kmeans.labels_
centroids = kmeans.cluster_centers_The n_init=10 parameter runs the algorithm 10 times with different random initializations and selects the best result, helping avoid poor local optima. The algorithm converged quickly, indicating well-separated clusters.
Cluster assignments: [3 3 0 1 3 1 2 1 0 2] Centroids shape: (4, 2) Number of iterations to converge: 4 Inertia (within-cluster sum of squares): 37.22
These results indicate good algorithm performance. The number of iterations shows how quickly the algorithm converged, which provides insight into cluster separation. The inertia value provides a baseline for comparing different clustering solutions, and the centroids shape confirms we have 4 cluster centers in our 2-dimensional feature space.
Step 3: Clustering Quality Evaluation
We'll evaluate the clustering performance using multiple metrics to assess the results from different angles.
from sklearn.metrics import adjusted_rand_score
# Calculate quality metrics
silhouette_avg = silhouette_score(X_scaled, cluster_labels)
ari_score = adjusted_rand_score(y_true, cluster_labels)Silhouette Score: 0.698 Adjusted Rand Index: 0.982 Inertia: 37.22
- Silhouette Score: Values above 0.6 indicate strong, well-separated clusters. This score shows that data points are much closer to their own cluster than to other clusters.
- Adjusted Rand Index: Values close to 1.0 show near-perfect agreement with true labels, indicating the algorithm successfully recovered the underlying cluster structure.
- Inertia: The within-cluster sum of squares provides a baseline for comparison with other k values.
Step 4: Finding the Optimal Number of Clusters
In practice, we often don't know the true number of clusters. Let's demonstrate how to determine the optimal k using systematic evaluation.
# Evaluate different numbers of clusters
k_range = range(1, 11)
inertias = []
silhouette_scores = []
for k in k_range:
kmeans_temp = KMeans(n_clusters=k, random_state=42, n_init=10)
kmeans_temp.fit(X_scaled)
inertias.append(kmeans_temp.inertia_)
# Silhouette score requires at least 2 clusters
if k > 1:
silhouette_scores.append(
silhouette_score(X_scaled, kmeans_temp.labels_)
)
else:
silhouette_scores.append(0)K values: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] Inertias: ['600.00', '324.65', '88.78', '37.22', '33.29', '29.71', '25.82', '23.03', '21.27', '19.24'] Silhouette scores: ['0.000', '0.526', '0.691', '0.698', '0.591', '0.497', '0.400', '0.340', '0.337', '0.365']
The results clearly show that k=4 is optimal:
- Inertia decreases monotonically with increasing k, but the rate of decrease slows after k=4, indicating diminishing returns
- Silhouette score peaks at k=4, confirming this as the best choice for cluster separation
- Performance degrades for k>4 as we over-cluster the data, creating artificial subdivisions that don't reflect the true structure
Step 5: Visualize the Results
Let's create a visualization that shows our clustering results and validation methods. This will help us understand both the clustering quality and the effectiveness of our optimization techniques.
You can see how the visualizations confirm our numerical results:
- Perfect cluster separation in the original data makes this an ideal test case
- K-means successfully recovers the true cluster structure with minimal misclassification
- Clear elbow at k=4 in the WCSS plot validates our choice of optimal clusters
- Peak silhouette score at k=4 provides additional confirmation of optimal clustering
Step 5: Real-World Example - Iris Dataset
Now let's apply K-means to a real dataset to demonstrate practical considerations. The Iris dataset contains measurements of three flower species.
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
# Load and preprocess the Iris dataset
iris = load_iris()
X_iris = iris.data
y_iris = iris.target
# Standardize features - essential for distance-based clustering
scaler_iris = StandardScaler()
X_iris_scaled = scaler_iris.fit_transform(X_iris)
# Apply K-means with 3 clusters (known number of species)
kmeans_iris = KMeans(n_clusters=3, random_state=42, n_init=10)
kmeans_iris.fit(X_iris_scaled)
centroids_iris_original = scaler_iris.inverse_transform(
kmeans_iris.cluster_centers_
)
# Evaluate clustering quality
iris_silhouette = silhouette_score(X_iris_scaled, kmeans_iris.labels_)
iris_ari = adjusted_rand_score(y_iris, kmeans_iris.labels_)Iris Dataset Results: Silhouette Score: 0.460 Adjusted Rand Index: 0.620 Cluster sizes: [53 50 47]
The results show moderate clustering performance:
- Silhouette Score: Moderate quality, indicating some overlap between clusters. This is lower than our synthetic data because real-world data has natural variation and boundary cases.
- Adjusted Rand Index: Strong agreement with true species labels, showing the algorithm captures the main species structure despite some misclassification.
- Uneven cluster sizes: The algorithm finds slightly different cluster sizes than the true 50-50-50 distribution.
Step 6: Optimizing k for Real Data
Let's determine the optimal number of clusters for the Iris dataset using both elbow method and silhouette analysis.
# Evaluate different k values for Iris dataset
k_range = range(1, 8)
inertias = []
silhouette_scores = []
for k in k_range:
kmeans_temp = KMeans(n_clusters=k, random_state=42, n_init=10)
kmeans_temp.fit(X_iris_scaled)
inertias.append(kmeans_temp.inertia_)
if k > 1:
silhouette_scores.append(
silhouette_score(X_iris_scaled, kmeans_temp.labels_)
)
else:
silhouette_scores.append(0)Iris Dataset Analysis Results: ================================================== K | Inertia | Silhouette Score ------------------------------ 1 | 600.00 | 0.000 2 | 222.36 | 0.582 3 | 139.82 | 0.460 4 | 114.09 | 0.387 5 | 90.93 | 0.346 6 | 81.54 | 0.317 7 | 72.63 | 0.320 Optimal k by Silhouette Score: 2 True number of species: 3 Note: While k=2 has the highest silhouette score, k=3 matches the true number of Iris species.
This reveals an important insight: the optimal k depends on the evaluation metric:
- Silhouette analysis suggests k=2 (0.681 score) - this groups the more similar Versicolor and Virginica together, which is mathematically optimal but ignores biological reality
- True biological reality is k=3 - three distinct species exist, and k=3 still performs well (0.553 score)
- Domain knowledge matters - sometimes the mathematically optimal solution doesn't align with real-world understanding
Step 8: Final Visualization of Iris Results
Let's visualize the final clustering results on the Iris dataset to see how well K-means performs on real data.
You can see how the visualizations reveal the clustering challenges in real data:
- Setosa is perfectly separated - the algorithm correctly identifies this distinct species
- Versicolor and Virginica show overlap - some misclassification occurs in the boundary region
- Overall good performance - the algorithm captures the main species structure despite natural variation
Key Parameters
Understanding the key parameters and methods is important for effective K-means implementation. The main aspects are:
Parameter Descriptions
n_clusters: The number of clusters k to form - this is the most important parameter that needs to be specified before running the algorithminit: Initialization method ('k-means++' is default and recommended for better convergence than random initialization)n_init: Number of times the algorithm runs with different centroid seeds (default 10, higher values reduce risk of poor local optima)max_iter: Maximum iterations per run (default 300, usually sufficient for convergence)random_state: Seed for reproducible results, including during research and debuggingalgorithm: The algorithm variant ('lloyd' is the standard K-means, 'elkan' is faster for well-separated clusters)
Best Practices
To achieve the best results with K-means clustering, it is important to follow several best practices. First, standardize your data before applying K-means, as the algorithm is highly sensitive to the scale of features due to its reliance on distance calculations. Using multiple initializations (setting n_init greater than 1) is recommended to avoid poor local optima and produce more consistent clustering outcomes. When evaluating your clustering results, do not rely solely on inertia, since this metric decreases monotonically as the number of clusters increases. Instead, combine inertia with another evaluation method such as silhouette analysis.
It is also advisable to use the k-means++ initialization method, as it typically leads to better clustering performance compared to random initialization. For reproducibility, set the random_state parameter, which ensures that your results are consistent across different runs. When assessing the quality of your clusters, compare the silhouette score and inertia, then interpret them using your domain knowledge. Finally, consider the characteristics of your data: K-means performs best when clusters are roughly spherical and of similar size.
Common Pitfalls
Some common pitfalls can undermine the effectiveness of K-means clustering if not carefully addressed. One frequent mistake is neglecting to standardize the data before clustering. Since K-means relies on distance calculations, features with larger scales can disproportionately influence the results, leading to misleading clusters. Another issue arises when the algorithm is run with only a single initialization; this increases the risk of converging to a poor local optimum, as K-means is sensitive to the initial placement of centroids.
Selecting the number of clusters (k) based solely on inertia is also problematic, as inertia will decrease as more clusters are added, potentially encouraging overfitting. It is important to combine inertia with other evaluation metrics and domain knowledge to make an informed choice. Ignoring domain expertise can result in clusters that are mathematically optimal but do not align with real-world categories or business objectives. Finally, failing to validate clustering results using multiple metrics and visualizations can obscure issues such as poorly separated clusters or misclassifications.