Dimensionality Reduction
A real dataset often has twenty variables, or two hundred, not two. You can't plot that many, models struggle with them (the curse of dimensionality), and many of them are redundant. Dimensionality reduction compresses many columns into two or three you can plot, while keeping as much of the structure as it can. There are two main families, and you can trust them in different ways: PCA, a linear method whose output you can measure, and t-SNE / UMAP, nonlinear maps that you should only look at.
PCA: the linear workhorse
You met principal component analysis in Factor Analysis & PCA. Here it is used to reduce the number of variables. PCA finds new axes (principal components) that are straight-line combinations of your original variables. The first is chosen to capture the most variance in the data, the second the most of what is left (at right angles to the first), and so on. Keep the top two or three and you have a low-dimensional projection, or "shadow", of the data that keeps as much of the variance as possible.
Because every component is a weighted sum of the originals, PCA is linear and reversible, and it never exaggerates distances. Points far apart in the plot were far apart in the full space, and the axes have an interpretation (the loadings). The cost is that PCA can only rotate and flatten the cloud. It cannot unfold curved structure. Give it two rings looped through each other and no flat shadow can separate them, because only a bend separates them, and a linear map cannot undo a bend.
🔻 Squash 3-D down to 2-D three ways
The same cloud: ring A, ring B (looped through it), and a stray blob. Drag the 3-D view to rotate and see the interlock. PCA gives the best flat shadow, but the rings still overlap, because a linear map cannot unfold a loop. A nonlinear embedding separates them. The three panels below show the same nonlinear method at three perplexity settings, and the picture changes each time (illustrative).
t-SNE and UMAP: seeing versus measuring
When the interesting structure is a bend, you need a nonlinear method. t-SNE and UMAP don't rotate the data. They build a new 2-D layout that tries to keep each point near the neighbors it had in the full space, and they are very good at making clusters visible. Use them for seeing structure you would never spot in a table.
But the freedom that lets them reveal clusters also lets them distort geometry. In a t-SNE or UMAP plot:
- Cluster sizes are meaningless. A tight blob and a sprawling one may hold the same variance, because the method inflates dense regions and shrinks sparse ones.
- Distances between clusters are meaningless. Two clumps far apart are not "more different" than two clumps that are close. Only the grouping within neighborhoods can be trusted.
- The settings change the picture. t-SNE's perplexity (and UMAP's
n_neighbors) can split one real cluster into several or merge several into one, as the trio in the widget shows. The layout is also random from run to run.
So never run a statistic on embedding coordinates, never report a gap or a cluster size as a finding, and treat any cluster the map suggests as a hypothesis to confirm in the original feature space. Use PCA when you need to measure, and t-SNE/UMAP when you need to look.
Putting the components to work
PCA is useful beyond plotting. Every case gets a score on every component, and those scores are ordinary variables you can use in a later model. The components are at right angles to each other, so their scores are uncorrelated by construction. That makes them a standard remedy for multicollinearity.
Take four predictors, three of them near-copies of one shared influence. In the raw model those three have variance inflation factors close to 7 and standard errors around 0.16, the usual signs of unstable estimates. Replace the four predictors with their four principal components and every VIF is exactly 1.000. The largest correlation between any two component scores is about 0.0000000000000014, which is zero to the limits of floating-point arithmetic.
PCA has one weakness here: it never looks at your outcome. Components are ranked by how much predictor variance each captures, not by how much of y each explains, and the two orderings need not agree. In the same example the first component holds 70.7% of the predictor variance and correlates .54 with the outcome. The second holds only 24.4% of the variance but correlates .47, nearly as much. Keeping all four components reproduces the original model's R² exactly (.5238 both ways), so you gain nothing until you drop some. And when you drop the components with the least variance, you may drop one that mattered for y.
Why it matters: reducing dimensions lets you visualize, denoise and simplify high-dimensional data, but the two families deserve different kinds of trust. PCA is linear and measurable. Its axes mean something and its plots never exaggerate distances, so it is your default for compression and for plots you will measure from, but it cannot unfold curves. t-SNE and UMAP can unfold curves and reveal clusters no linear method can, but the sizes, gaps and shapes in their plots come from the algorithm and its perplexity, not from the data. Look with the nonlinear maps, measure with PCA, and check every cluster before you believe it.
Common questions
Is PCA the same as factor analysis?
They are closely related and often give similar-looking results, but they answer different questions. PCA is a purely mathematical rotation. It turns your variables into uncorrelated components ordered by how much total variance each captures, with no model of why the variables move together, so it is best thought of as data compression. Exploratory factor analysis assumes that unobserved latent factors cause the observed correlations, and it tries to estimate those factors while setting aside each variable's unique noise. As a rule of thumb, use PCA when you want to reduce dimensions or remove the correlation between predictors, and factor analysis when you are hypothesizing underlying constructs (like 'verbal ability') behind a battery of measures.
Can I run statistics on t-SNE or UMAP coordinates?
No. A t-SNE or UMAP embedding is built to preserve local neighborhoods for visualization, and to do that it distorts global geometry. Distances between clusters, the sizes of clusters, and even the number of apparent clumps all change with the perplexity or n_neighbors setting and with the random seed. So the axes have no units and the gaps mean nothing. A t-test, a correlation or a clustering run on the 2-D coordinates produces numbers that describe the algorithm, not your data. Use the embedding only to spot structure worth investigating, then go back to the original feature space (or a PCA) to measure anything.
Do I need to scale my variables before PCA?
Almost always, yes, and forgetting to is the most common way to get a meaningless first component. PCA looks for directions of maximum variance, and variance depends on units. Measure income in dollars rather than thousands and its variance grows by a factor of a million, so PC1 becomes essentially the income axis whatever the other variables are doing. Standardizing every variable first (mean 0, SD 1) puts them on the same scale. That is the same as running PCA on the correlation matrix rather than the covariance matrix. The exception is when your variables are already in the same meaningful unit and you want the high-variance ones to dominate: repeated measurements of one quantity, or pixel intensities in an image. Unless you can say why the raw units should decide the answer, scale. For how many components to keep, see Factor Analysis / PCA.