Dimensionality Reduction
Real datasets rarely have two variables — they have twenty, or two hundred. You can't plot that, models choke on it (the curse of dimensionality), and much of it is redundant. Dimensionality reduction squeezes many columns down to a plottable two or three while keeping as much of the structure as it can. Two families dominate, and they demand opposite kinds of trust: PCA, the linear workhorse you can measure, and t-SNE / UMAP, the nonlinear maps you should only look at.
PCA: the linear workhorse
You met principal component analysis in Factor Analysis & PCA; here it earns its keep as a reduction tool. PCA finds new axes (principal components) that are straight-line combinations of your original variables, chosen so the first captures the most variance in the data, the second the most of what's left (at right angles to the first), and so on. Keep the top two or three and you have a faithful, low-dimensional shadow of the data.
Because every component is just a weighted sum of the originals, PCA is linear, reversible, and honest about distances: points far apart in the plot were far apart in the full space, and the axes mean something you can interpret (loadings). The price of that honesty is rigidity — PCA can only rotate and flatten the cloud. It cannot unfold curved structure. Hand it two rings looped through each other and no flat shadow will ever pull them apart, because the separation lives in a bend that a linear map can't undo.
🔻 Squash 3-D down to 2-D three ways
The same cloud: ring A, ring B (looped through it), and a stray blob. Drag the 3-D view to rotate and see the interlock. PCA is the best flat shadow — yet the rings still overlap, because linear maps can't unfold a loop. A nonlinear embedding untangles them. The trio below shows the same nonlinear method at three perplexity settings — the picture changes every time (illustrative).
t-SNE and UMAP: seeing versus measuring
When the interesting structure is a bend, you need a nonlinear method. t-SNE and UMAP don't rotate the data; they build a fresh 2-D layout that tries to keep each point near the neighbors it had in the full space, and they are dazzlingly good at making clusters pop out. That's their job and their gift: seeing structure you'd never spot in a table.
But the same freedom that reveals clusters makes the map a liar about geometry. In a t-SNE or UMAP plot:
- Cluster sizes are meaningless. A tight blob and a sprawling one may hold the same variance — the method inflates dense regions and shrinks sparse ones.
- Distances between clusters are meaningless. Two clumps sitting far apart aren't "more different" than two clumps that are close; only the within-neighborhood grouping is trustworthy.
- The settings change the story. t-SNE's perplexity (and UMAP's
n_neighbors) can split one real cluster into several or merge several into one — as the trio in the widget shows. The layout is also random from run to run.
So the rule is blunt: never run a statistic on embedding coordinates, never report a gap or a cluster size as a finding, and treat any cluster the map suggests as a hypothesis to confirm back in the real feature space. Use PCA when you need to measure; use t-SNE/UMAP when you need to look.
Putting the components to work
Measuring with PCA means more than plotting it. Every case gets a score on every component, and those scores are ordinary variables you can carry into a later model. Since the components are built at right angles to each other, their scores arrive uncorrelated by construction, which makes them a standard remedy for multicollinearity.
How completely uncorrelated? Take four predictors, three of them near-copies of one shared influence. Fitted raw, those three carry variance inflation factors close to 7 and standard errors around 0.16, the familiar unstable picture. Replace them with their four principal components and every VIF is exactly 1.000, because the largest correlation left between any two component scores is about 0.0000000000000014, which is zero to the limits of floating-point arithmetic.
PCA has one blind spot, and it is why this is a tool rather than a reflex: it never looks at your outcome. Components are ranked by how much predictor variance each captures, not by how much of y each explains, and those two orderings need not agree. In that same example the first component holds 70.7% of the predictor variance and correlates .54 with the outcome, while the second holds only 24.4% and correlates .47, very nearly as much. Keeping all four components reproduces the original model's R² to the last digit (.5238 both ways), so you gain nothing until you drop some, and dropping is exactly where a component that mattered for y can go out with the trash.
Why it matters: Reducing dimensions is how you visualize, denoise, and de-clutter high-dimensional data — but the two toolkits earn different kinds of trust. PCA is linear and measurable: its axes mean something and its distances are real, so it's your default for compression and honest plotting, at the cost of never unfolding curves. t-SNE and UMAP unfold beautifully and reveal clusters no linear method can, but their sizes, gaps, and shapes are artifacts of the algorithm and its perplexity, not the data. See with the nonlinear maps; measure with PCA; verify every cluster before you believe it.
Common questions
Is PCA the same as factor analysis?
They are close cousins and often give similar-looking results, but they answer different questions. PCA is a purely mathematical rotation: it repackages your variables into uncorrelated components ordered by how much total variance each captures, with no model of why the variables move together — it is best thought of as data compression. Exploratory factor analysis, by contrast, assumes there are unobserved latent factors that cause the observed correlations, and it tries to estimate those factors while setting aside each variable's unique noise. Rule of thumb: reach for PCA when you want to reduce dimensions or de-correlate predictors, and for factor analysis when you are hypothesizing underlying constructs (like 'verbal ability') behind a battery of measures.
Can I run statistics on t-SNE or UMAP coordinates?
No. A t-SNE or UMAP embedding is built to preserve local neighborhoods for the purpose of visualization, and in doing so it deliberately distorts global geometry: distances between clusters, the sizes of clusters, and even the number of apparent clumps all change with the perplexity or n_neighbors setting and with the random seed. So the axes have no units, the gaps mean nothing, and running a t-test, a correlation, or a clustering on the 2-D coordinates produces numbers that describe the algorithm, not your data. Use the embedding only to spot structure worth investigating, then go back to the original feature space (or a PCA) to measure anything.
Do I need to scale my variables before PCA?
Almost always, yes, and forgetting is the commonest way to get a meaningless first component. PCA hunts for directions of maximum variance, and variance carries units: measure income in dollars rather than thousands and its variance grows by a factor of a million, so PC1 becomes essentially the income axis no matter what the other variables are doing. Standardizing every variable first (mean 0, SD 1) puts them on equal footing, which is the same operation as running PCA on the correlation matrix rather than the covariance matrix. The exception is when your variables are already in the same meaningful unit and you want the high-variance ones to dominate: repeated measurements of one quantity, or pixel intensities in an image. If you cannot articulate why the raw units should decide the answer, scale. For how many components to then keep, see Factor Analysis / PCA.