Factor Analysis / PCA
A personality survey might ask 50 questions, but they aren't 50 independent things: answers cluster, because they tap a handful of underlying traits. Factor analysis and Principal Component Analysis (PCA) are the tools that find those few hidden dimensions hiding inside many correlated variables. They turn "50 columns" into "5 things that actually matter."
The core idea: directions of variance
PCA looks for the directions in which the data varies the most. The first principal component is the single line that captures as much of the spread as possible; the second is the best remaining direction perpendicular to the first, and so on. When variables are highly correlated, almost all the variance lines up along one direction, so one component can stand in for many.
🎮 Watch the Components Find the Cloud
A cloud of two correlated variables. The long arrow is PC1 (most variance), the short one PC2. Crank up the correlation and watch the cloud stretch into a line and one component swallow nearly all the variance.
That last readout is the one to watch. Two variables technically live in 2D, but when they're strongly correlated the cloud is really a slightly fattened line, close to one-dimensional. PCA quantifies that: if PC1 explains 95% of the variance, you can describe the data with one number per point instead of two and lose almost nothing. Scale that up from 2 variables to 50 and you have serious dimensionality reduction, which is where the same machinery reappears in machine learning.
How many components to keep
The usual aids are the scree plot (variance explained by each component — look for the "elbow" where it levels off), the Kaiser criterion (keep components with eigenvalue > 1), and simply keeping enough to reach a target like 80–90% of total variance. There's judgment involved; interpretability matters as much as the numbers.
PCA vs. factor analysis
They look similar but answer different questions. PCA is descriptive: it repackages the variance into uncorrelated components, no model assumed. Factor analysis is a model: it assumes real, unobserved latent factors (like "extraversion") that cause the observed responses, and separates shared variance from each variable's unique noise. Use PCA to compress; use factor analysis to argue that a hidden trait exists.
Is the matrix worth factoring at all?
Before any of this, software asks whether your variables are correlated enough to have shared structure to find. Two screens come as standard, and both are worth understanding rather than ticking.
KMO (the Kaiser-Meyer-Olkin measure of sampling adequacy) compares the ordinary correlations between your variables with their partial correlations, the ones left after every other variable has been held constant. Shared structure means the plain correlations survive that adjustment and KMO climbs toward 1; variables that only correlate in unrelated pairs push it down. Kaiser's labels run from marvelous above .90 through middling in the .70s to unacceptable below .50, and .60 is the usual working floor. A clean two-factor dataset scores around .91; pure noise scores about .50.
Bartlett's test of sphericity asks a much weaker question: could your correlation matrix have come from variables that are all mutually uncorrelated? Treat a significant result as a formality rather than an achievement. With 20 variables and 100 people, true correlations of just .10 already give p = .00005, and at 300 people the same tiny correlations give p below 10⁻¹⁶. A non-significant Bartlett means stop; a significant one means almost nothing on its own, so read KMO and the scree plot for the real answer.
Reading the result
Each component has loadings: how strongly each original variable contributes to it. You interpret a component by looking at which variables load heavily on it: if questions about parties, talking, and meeting strangers all load on one factor, you might name it "sociability." Naming factors is part science, part art.
Why it matters: PCA and factor analysis are everywhere data is wide: psychometrics, genomics, image compression, recommender systems. They're how you find structure and signal when you have far more variables than you can reason about one at a time. In psychometrics especially they sit right next to reliability and validity: a factor structure is the evidence that a scale measures the one thing it claims to.
Common questions
How many factors or components should I keep?
Triangulate: the scree plot's elbow (keep components before the curve flattens), parallel analysis (keep factors whose eigenvalues beat those from random data — the most defensible modern criterion), and interpretability (can you name each factor?). The old Kaiser rule (eigenvalue > 1) is simple but notoriously over-extracts. When criteria disagree, favor the solution that makes theoretical sense.
What is rotation, and should I choose varimax or oblimin?
Extraction finds a set of factors that fits; rotation spins those factors to a position that a human can name, without changing how well the model fits or how much total variance it explains. The choice is a claim about your constructs. Varimax is orthogonal: it forces the factors to be uncorrelated, which gives the cleanest-looking loading table. Oblimin and its relatives are oblique: they let the factors correlate, and then report how much. In psychology the honest default is usually oblique, because real traits do correlate, and an oblique rotation will tell you so. A useful habit is to run oblimin first and inspect the factor correlations: if they all come out near zero, varimax was defensible after all and you now have evidence rather than an assumption. Note that oblique solutions print two tables, and it is the pattern matrix you interpret.
What is a factor loading, and what value is good?
The correlation-like weight tying an observed variable to a factor — how much that item "belongs." Conventional floors: |loading| ≥ .40 for a variable to count toward a factor (≥ .70 is excellent), while items loading ≥ .30–.40 on multiple factors (cross-loadings) make interpretation murky and are often revised or dropped in scale development.