Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

k-Nearest Neighbors › Visualizing k-Nearest Neighbors

Michael BrenndoerferOctober 26, 20268 min read

Machine Learning from ScratchAll chapters

Part 7 of 15Supervised Learning Beyond Linear Models

Chapter 53k-Nearest Neighbors › Visualizing k-Nearest Neighbors

Reading section 5 of 10 · Visualizing k-Nearest Neighbors · 8 minSection 5 of 10 · 8 min

Section 5 of 10 · 8 min read

k-Nearest Neighbors › Visualizing k-Nearest NeighborsSection view · Sections in this chapter5 / 10 ↓

Sections in this chapter10

Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Notes & commentsLeave notes and join the discussionSelection toolbarShow comment actions when selecting text. Turn off for the browser menu only.Comment indicatorsShow small counts beside annotated passagesPassage highlightsGently shade annotated text selections

The clearest way to understand kNN is to see its decision boundary change with $k$ and with feature scaling. We generate a two-dimensional, mildly noisy classification problem and draw the decision region for $k = 1$ , $k = 5$ , and $k = 25$ on a grid of query points. The following cell defines the majority-vote classifier before computing those decision regions.

PythonIn [5]
CollapseExpandrng = np.random.default_rng(7)

# Synthetic two-class problem
X_viz = np.vstack(
    [
        rng.normal(loc=(1.0, 1.0), scale=0.9, size=(60, 2)),
        rng.normal(loc=(2.5, 2.5), scale=0.9, size=(60, 2)),
    ]
)
y_viz = np.array([0] * 60 + [1] * 60)
X_train, y_train = X_viz, y_viz

# Query grid over the feature space
x_min, x_max = X_train[:, 0].min() - 1.0, X_train[:, 0].max() + 1.0
y_min, y_max = X_train[:, 1].min() - 1.0, X_train[:, 1].max() + 1.0
gx, gy = np.meshgrid(
    np.linspace(x_min, x_max, 200), np.linspace(y_min, y_max, 200)
)
grid_points = np.column_stack([gx.ravel(), gy.ravel()])


def knn_classification_predict(X_train, y_train, X_query, k):
    """Brute-force kNN for classification (majority vote)."""
    X_train = np.asarray(X_train)
    X_query = np.asarray(X_query)
    y_train = np.asarray(y_train)
    if not isinstance(k, (int, np.integer)) or not 1 <= k <= len(X_train):
        raise ValueError(
            "k must be an integer between 1 and the training-set size"
        )

    diffs = X_query[:, None, :] - X_train[None, :, :]
    dists = np.sqrt((diffs**2).sum(axis=2))
    nn_idx = np.argsort(dists, axis=1, kind="stable")[:, :k]
    neighbor_labels = y_train[nn_idx]

    classes = np.unique(y_train)
    preds = []
    for row in neighbor_labels:
        counts = np.array([(row == c).sum() for c in classes])
        preds.append(classes[counts.argmax()])
    return np.array(preds)


boundary_ks = [1, 5, 25]
boundary_preds = {
    k: knn_classification_predict(X_train, y_train, grid_points, k)
    for k in boundary_ks
}
boundary_surfaces = {
    k: predictions.reshape(gx.shape)
    for k, predictions in boundary_preds.items()
}
viz_class_masks = {cls: y_train == cls for cls in [0, 1]}

# Fixed axis limits shared by the three boundary figures.
viz_xlim = (float(gx.min()), float(gx.max()))
viz_ylim = (float(gy.min()), float(gy.max()))

::: {layout-ncol=3}

VisualizationOut [6]
CollapseExpand

A single two-feature scatter plot with class-specific markers and the k=1 decision region. — kNN classification decision boundary for k=1 on the seeded two-class dataset. Shaded regions show the majority-vote assignment across the feature space, and markers give the labeled training points. The boundary meanders through the data and carves out small pockets around individual points, showing that a single neighbor memorizes local noise.

VisualizationOut [7]
CollapseExpand

A single two-feature scatter plot with class-specific markers and the k=5 decision region. — kNN classification decision boundary for k=5 on the seeded two-class dataset. With five voters the decision region is smoother than at k=1 while still following the broad shape of the two blobs, showing the fitted region when each query combines five neighbor labels.

VisualizationOut [8]
CollapseExpand

A single two-feature scatter plot with class-specific markers and the k=25 decision region. — kNN classification decision boundary for k=25 on the seeded two-class dataset. With 25 voters the region is nearly linear and ignores local irregularities, showing how broader neighborhoods suppress local detail.

:::

The three neighboring figures compare the effect of $k$ . At $k=1$ the boundary meanders through the data, carving out small pockets around individual points: at these distinct training inputs, 1NN reproduces the observed labels. At this small sample size, the visual flexibility of k=1 does not by itself establish an accuracy advantage, so on this data the added complexity may buy little. At $k=5$ the boundary is smoother but still follows the broad shape of the two blobs. At $k=25$ the boundary is nearly linear and ignores local irregularities; nearby points that fall on the "wrong" side of the smooth curve are outvoted. These pictures show how the fitted boundary changes with $k$ . Small neighborhoods can be sensitive to individual observations, while larger neighborhoods can smooth away local detail. Measuring prediction variance requires a specified repeated-sampling experiment.

To see how scaling can distort kNN, we can repeat the picture with one feature magnified by a factor of 50, as though it had not been standardized. Distances are then dominated by the magnified feature, and neighbors are chosen almost entirely along that axis.

PythonIn [9]
CollapseExpand# Rescale the first coordinate for both stored points and queries.
X_train_bad = X_train * np.array([50.0, 1.0])
grid_bad = grid_points * np.array([50.0, 1.0])
preds_ok = knn_classification_predict(X_train, y_train, grid_points, 5)
preds_bad = knn_classification_predict(X_train_bad, y_train, grid_bad, 5)
scaling_surface_ok = preds_ok.reshape(gx.shape)
scaling_surface_bad = preds_bad.reshape(gx.shape)
scaling_x_grid = gx * 50.0

::: {layout-ncol=2}

VisualizationOut [10]
CollapseExpand

A single two-feature classification plot. Comparable feature scales. Class markers and a shaded decision region identify the two classes. — k=5 classification using the original comparable feature scales. The two classes form overlapping blobs, and the decision region roughly bisects them because both features enter the distance in their original, comparable units, providing the reference picture for the magnified-scale case.

VisualizationOut [11]
CollapseExpand

A single two-feature classification plot. Feature 1 magnified 50x. Class markers and a shaded decision region identify the two classes. — k=5 classification with the first feature multiplied by 50 for both training points and queries. Multiplying the first feature by 50 multiplies its own squared-distance contribution by 2500 while leaving the second feature unchanged. In this sample, that change produces narrow vertical decision bands.

:::

In the right figure, the decision regions form narrow vertical bands. Multiplying the first feature by 50 multiplies its own squared-distance contribution by $50^2 = 2500$ , while leaving the second feature unchanged. On this sample, neighbors are then chosen largely by similarity in the first feature. The distance depends on the scales on which features are measured. Use scaling when needed to prevent units from imposing unintended feature weights.

These figure groups show the behavior of $k$ and of feature scaling. The implementation below turns the same mechanics into auditable NumPy code and compares it with scikit-learn.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

Comments

Add to the conversation

Stay up to date