Interactivet-SNE: Complete Guide to Dimensionality Reduction & High-Dimensional Data Visualization

t-SNE

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a powerful dimensionality reduction technique that excels at visualizing high-dimensional data by preserving local neighborhood structures. Unlike linear methods such as PCA, t-SNE is specifically designed to reveal non-linear relationships and complex data structures that might be hidden in high-dimensional spaces. The method works by modeling the probability distribution of pairwise similarities in the original high-dimensional space and then finding a low-dimensional representation that preserves these similarities as closely as possible.

The key insight behind t-SNE is that it uses different probability distributions for the high-dimensional and low-dimensional spaces. In the high-dimensional space, it uses Gaussian distributions to model similarities, while in the low-dimensional space, it uses the heavier-tailed t-distribution. This asymmetry helps prevent the "crowding problem" where points would otherwise cluster too tightly in the center of the low-dimensional space, making it particularly effective for visualization tasks.

t-SNE is especially valuable for exploratory data analysis, where we want to understand the underlying structure of our data before applying more complex machine learning algorithms. It has become a standard tool in fields ranging from genomics and neuroscience to computer vision and natural language processing, where high-dimensional data visualization is crucial for gaining insights.

Advantages

t-SNE excels at revealing non-linear structures and complex data relationships that linear dimensionality reduction methods like PCA cannot capture. The method is particularly effective at separating distinct clusters and identifying outliers, making it valuable for exploratory data analysis. The heavy-tailed t-distribution used in the low-dimensional space helps prevent the crowding problem, allowing for better separation of data points and more interpretable visualizations.

The method is relatively robust to different types of noise and can handle various data types effectively. It preserves local neighborhood structures well, which means that points close in the original high-dimensional space typically remain close in the low-dimensional visualization. This local structure preservation makes t-SNE useful for understanding the fine-grained structure of complex datasets.

Disadvantages

One of the most significant limitations of t-SNE is that it is computationally expensive, with time complexity of O(n²), making it impractical for very large datasets (typically limited to datasets with fewer than 10,000 points). The method is also stochastic, meaning that different runs can produce different results, which can make interpretation challenging and requires multiple runs to ensure consistency.

t-SNE is primarily a visualization tool and should not be used for feature reduction in machine learning pipelines, as it doesn't preserve global structure and can be sensitive to hyperparameters. The method requires careful tuning of the perplexity parameter, which controls the effective number of neighbors and significantly impacts the results. Additionally, t-SNE can sometimes create misleading visualizations where apparent clusters in the low-dimensional space don't correspond to meaningful groups in the original data.

Building the Mathematical Foundation

To understand how t-SNE works, we need to follow its core logic: the algorithm tries to preserve relationships between nearby points when moving from high dimensions to low dimensions. Instead of trying to preserve exact distances, t-SNE preserves similarities. Think of it as keeping friends close together rather than measuring exact distances between people in a room.

The algorithm operates in three main steps. First, it measures how similar points are in the original high-dimensional space. Second, it creates a low-dimensional representation and measures similarities there. Third, it adjusts the low-dimensional positions until the two sets of similarities match as closely as possible.

Measuring similarities in high-dimensional space

We start with our original data points $x_i$ and $x_j$ in high-dimensional space. The first step is straightforward: compute how far apart these points are using squared Euclidean distance.

$d_{ij} = ||x_i - x_j||^2$

where:

• $x_i$: the $i$-th data point in the original high-dimensional space
• $x_j$: the $j$-th data point in the original high-dimensional space
• $d_{ij}$: squared Euclidean distance between points $i$ and $j$
• $||\cdot||^2$: squared L2 norm (sum of squared differences across all dimensions)

This gives us raw distances, but t-SNE doesn't work directly with distances. Instead, it converts them into probabilities that express similarity. Why probabilities? Because they naturally normalize the scale and make similarities relative rather than absolute. A distance of 5 units means something different if most points are separated by 100 units versus if most are separated by 1 unit.

To convert distances into similarities, we use a Gaussian (normal) distribution centered at each point. The conditional probability $p_{j|i}$ represents the probability that point $x_i$ would pick $x_j$ as its neighbor if neighbors were selected in proportion to their probability density under this Gaussian.

$p_{j|i} = \frac{\exp(-d_{ij}/2\sigma_i^2)}{\sum_{k \neq i} \exp(-d_{ik}/2\sigma_i^2)}$

where:

• $p_{j|i}$: conditional probability that point $i$ picks point $j$ as its neighbor
• $\sigma_i$: bandwidth of the Gaussian kernel centered at point $i$ (controls the effective neighborhood size)
• $\exp(\cdot)$: exponential function (converts distances to similarities)
• $\sum_{k \neq i}$: sum over all points $k$ except point $i$ (normalization)

Let's break down what each part does:

• The numerator $\exp(-d_{ij}/2\sigma_i^2)$ gives higher values to closer points and lower values to distant points. The exponential ensures all values are positive.
• The variance $\sigma_i^2$ controls the width of the Gaussian. Larger $\sigma_i$ means we consider more distant neighbors.
• The denominator normalizes these values so they sum to 1, making them proper probabilities.

The variance $\sigma_i$ is different for each point, which is unusual but important. In dense regions of the data, we use smaller $\sigma_i$ to focus on very close neighbors. In sparse regions, we use larger $\sigma_i$ to capture the local structure. The perplexity parameter controls these variances indirectly. Higher perplexity means the Gaussian is wider, effectively considering more neighbors.

Now we have conditional probabilities $p_{j|i}$, but there's a problem: $p_{j|i}$ might not equal $p_{i|j}$. To fix this asymmetry, we symmetrize by averaging:

$p_{ij} = \frac{p_{j|i} + p_{i|j}}{2n}$

where:

• $p_{ij}$: symmetric joint probability of similarity between points $i$ and $j$
• $n$: total number of data points in the dataset
• The factor $2n$ ensures that $\sum_{i}\sum_{j} p_{ij} = 1$

This gives us a symmetric joint probability distribution where $p_{ij} = p_{ji}$. The division by $2n$ ensures all probabilities sum to 1 over all pairs. These symmetrized probabilities $p_{ij}$ represent our target: they encode which points should be close together in the visualization.

Measuring similarities in low-dimensional space

Next, we need to measure similarities in the low-dimensional space where we want to visualize our data. Let $y_i$ and $y_j$ be the low-dimensional coordinates we're trying to find. We could use the same Gaussian distribution here, but that creates a problem called crowding.

The crowding problem happens because high-dimensional spaces have much more "room" than low-dimensional spaces. Imagine 100 points evenly distributed in 50 dimensions. When we try to place them in 2 dimensions while keeping all pairwise distances the same, they'll be forced too close together in the center. Moderate distances in high dimensions become impossible to represent in low dimensions.

To solve this, t-SNE uses a heavy-tailed distribution in low dimensions: the Student t-distribution with one degree of freedom (also called the Cauchy distribution). The similarity between points $y_i$ and $y_j$ is:

$q_{ij} = \frac{(1 + ||y_i - y_j||^2)^{-1}}{\sum_{k} \sum_{l \neq k} (1 + ||y_k - y_l||^2)^{-1}}$

where:

• $q_{ij}$: similarity between points $i$ and $j$ in the low-dimensional space
• $y_i$: low-dimensional representation of point $i$ (coordinates we're optimizing)
• $y_j$: low-dimensional representation of point $j$
• $||y_i - y_j||^2$: squared Euclidean distance between points in low-dimensional space
• $(1 + ||y_i - y_j||^2)^{-1}$: Student t-distribution kernel with one degree of freedom
• $\sum_{k} \sum_{l \neq k}$: sum over all distinct pairs of points for normalization

The key difference from the Gaussian is $(1 + ||y_i - y_j||^2)^{-1}$ instead of $\exp(-||y_i - y_j||^2)$. The t-distribution has heavier tails, meaning it decreases more slowly as distance increases. This allows moderately distant points in low dimensions to still have reasonable similarity values, giving them room to spread out instead of crowding in the center.

The denominator again normalizes to make these proper probabilities that sum to 1.

Finding the right low-dimensional positions

Now we have two probability distributions: $p_{ij}$ representing similarities we want to preserve from the high-dimensional space, and $q_{ij}$ representing similarities in our current low-dimensional representation. We need to adjust the positions $y_i$ until these distributions match.

To measure how well they match, t-SNE uses the Kullback-Leibler (KL) divergence, a standard measure of how different two probability distributions are:

$C = \sum_{i} \sum_{j} p_{ij} \log \frac{p_{ij}}{q_{ij}}$

where:

• $C$: Kullback-Leibler divergence (cost function to minimize)
• $p_{ij}$: target similarity between points $i$ and $j$ in high-dimensional space
• $q_{ij}$: current similarity between points $i$ and $j$ in low-dimensional space
• $\log$: natural logarithm
• The sum is over all pairs of points

The KL divergence has useful properties for this task. When $p_{ij}$ is large (points should be close) but $q_{ij}$ is small (they're actually far apart), the log ratio is large and positive, heavily penalizing the mismatch. When $p_{ij}$ and $q_{ij}$ are both small (points should be far apart and are), the contribution is minimal. The cost function is asymmetric: it cares more about keeping close points together than about separating distant points, which aligns with t-SNE's goal of preserving local structure.

To minimize this cost, we use gradient descent. We need the derivative of the cost with respect to each low-dimensional position $y_i$:

$\frac{\partial C}{\partial y_i} = 4 \sum_j (p_{ij} - q_{ij})(y_i - y_j)(1 + ||y_i - y_j||^2)^{-1}$

where:

• $\frac{\partial C}{\partial y_i}$: gradient of the cost function with respect to position $y_i$
• The sum is over all other points $j$
• $(p_{ij} - q_{ij})$: difference between target and current similarities
• $(y_i - y_j)$: vector pointing from $y_j$ to $y_i$ in low-dimensional space
• $(1 + ||y_i - y_j||^2)^{-1}$: weighting factor from the t-distribution
• The factor of 4 comes from the derivative of the symmetrized KL divergence

This gradient has an elegant interpretation. For each pair of points:

• If $p_{ij} > q_{ij}$ (points should be closer than they are), the term $(p_{ij} - q_{ij})$ is positive. The gradient pulls $y_i$ toward $y_j$.
• If $p_{ij} < q_{ij}$ (points should be farther apart), the term is negative. The gradient pushes $y_i$ away from $y_j$.
• The magnitude of the force depends on how different the probabilities are.
• The factor $(1 + ||y_i - y_j||^2)^{-1}$ comes from the t-distribution and ensures the gradient doesn't explode for distant points.

The algorithm iteratively updates each $y_i$ by taking small steps in the direction opposite to this gradient, gradually moving points until the low-dimensional similarities $q_{ij}$ match the high-dimensional similarities $p_{ij}$ as closely as possible.

Key mathematical properties

Understanding a few mathematical properties helps explain t-SNE's behavior in practice.

Non-convexity and local minima: The cost function has multiple local minima because it's non-convex. Different random initializations can lead to different final configurations. This is why running t-SNE multiple times with different random seeds is important to ensure your visualization is stable.

Scale invariance: Multiplying all distances by a constant factor doesn't change the relative similarities. This means t-SNE results don't depend on the absolute scale of your data, only on the relative distances. However, the method is not rotation-invariant, so the orientation of your final visualization is arbitrary.

The role of perplexity: The perplexity parameter acts as a smoothness control. Lower values (5-15) create more localized neighborhoods, revealing fine-grained structure but potentially fragmenting the visualization. Higher values (30-50) create broader neighborhoods, emphasizing more global structure. The choice of perplexity significantly affects the result. A common default is 30, which typically balances local and global structure well for most datasets. The perplexity should generally be between 5 and 50, and you should try several values to see which gives the most interpretable visualization.

Visualizing t-SNE

Let's create visualizations to understand how t-SNE works with different types of data structures. We'll start with a simple example using synthetic data to demonstrate the key concepts.

Now let's examine how different perplexity values affect the t-SNE visualization:

Let's also visualize the effect of t-SNE on high-dimensional data by projecting it back to 2D:

Example

Let's work through a concrete example using a small dataset to demonstrate how t-SNE calculations work step by step. We'll use a simple 2D dataset with 8 points divided into two clusters, showing exactly how the formulas are applied in practice.

Step 1: Calculate pairwise distances

First, we compute the squared Euclidean distance $d_{ij} = ||x_i - x_j||^2$ for all pairs of points.

Step 2: Convert distances to probabilities

Next, we apply the Gaussian kernel formula to convert distances into conditional probabilities. For demonstration, we'll calculate $p_{j|0}$ (the probability that point 0 picks each other point as a neighbor):

$p_{j|i} = \frac{\exp(-d_{ij}/2\sigma_i^2)}{\sum_{k \neq i} \exp(-d_{ik}/2\sigma_i^2)}$

We use $\sigma = 1.0$ for simplicity (in practice, $\sigma$ is determined by the perplexity parameter).

Notice how points 1, 2, and 3 (which are close to point 0 in the first cluster) have higher probabilities, while points 4-7 (in the distant second cluster) have probabilities close to zero.

Step 3: Apply complete t-SNE algorithm

Now we apply the full t-SNE algorithm, which computes all pairwise probabilities, symmetrizes them, creates a low-dimensional representation, and iteratively optimizes the positions to minimize the KL divergence.

Implementation in Scikit-learn

Scikit-learn provides a robust implementation of t-SNE through the TSNE class. This implementation is optimized for performance and includes several important features for practical use. Let's walk through a complete example using the digits dataset, which contains images of handwritten digits.

Loading and preparing the data

We'll start by loading the digits dataset and examining its basic properties.

The digits dataset contains 1,797 samples, each representing an 8×8 pixel image (64 features total). With 10 digit classes (0-9), this is a perfect test case for t-SNE visualization, as it has enough dimensions to make visualization challenging but not so many points that computation becomes impractical.

Applying t-SNE

Before applying t-SNE, we standardize the data to ensure all features contribute equally to the distance calculations.

The computation takes a few seconds on this moderately-sized dataset. The KL divergence value indicates how well the low-dimensional representation preserves the high-dimensional similarities. Lower values are better, though there's no universal threshold for what constitutes a "good" value. The key is that the optimization has converged (the value stopped decreasing significantly).

Visualizing the results

Let's visualize the t-SNE embedding to see how well it separates the different digit classes.

The visualization shows clear separation between most digit classes, with distinct clusters forming for each number. This demonstrates that t-SNE successfully captured the underlying structure in the 64-dimensional space. Notice that some digits like 0, 1, and 6 form tight, well-separated clusters, while others like 3, 5, and 8 show more overlap, reflecting their visual similarity.

Exploring different perplexity values

Perplexity is the most important parameter in t-SNE. Let's see how different values affect the visualization.