Factor Analysis / PCA

A personality survey might ask 50 questions, but they aren't 50 independent things: answers cluster, because they tap a handful of underlying traits. Factor analysis and Principal Component Analysis (PCA) are the tools that find those few hidden dimensions hiding inside many correlated variables. They turn "50 columns" into "5 things that actually matter."

The core idea: directions of variance

PCA looks for the directions in which the data varies the most. The first principal component is the single line that captures as much of the spread as possible; the second is the best remaining direction perpendicular to the first, and so on. When variables are highly correlated, almost all the variance lines up along one direction, so one component can stand in for many.

That last readout is the one to watch. Two variables technically live in 2D, but when they're strongly correlated the cloud is really a slightly fattened line, close to one-dimensional. PCA quantifies that: if PC1 explains 95% of the variance, you can describe the data with one number per point instead of two and lose almost nothing. Scale that up from 2 variables to 50 and you have serious dimensionality reduction, which is where the same machinery reappears in machine learning.

How many components to keep

The usual aids are the scree plot (variance explained by each component — look for the "elbow" where it levels off), the Kaiser criterion (keep components with eigenvalue > 1), and simply keeping enough to reach a target like 80–90% of total variance. There's judgment involved; interpretability matters as much as the numbers.

Parallel analysis asks the same question against a fairer benchmark, and it is the retention rule most methodologists now recommend: simulate many random datasets with your number of variables and your number of people, average their eigenvalues, and keep only the components whose real eigenvalue beats the matching random one. That comparison exists because 1 is a low bar. With 12 variables and 100 respondents of pure noise, the largest eigenvalue averages 1.61, and on average 5.6 of the 12 clear 1.0, so the Kaiser criterion happily keeps components built entirely from sampling variation.

PCA vs. factor analysis

Is the matrix worth factoring at all?

Before any of this, software asks whether your variables are correlated enough to have shared structure to find. Two screens come as standard, and both are worth understanding rather than ticking.

KMO (the Kaiser-Meyer-Olkin measure of sampling adequacy) compares the ordinary correlations between your variables with their partial correlations, the ones left after every other variable has been held constant. Shared structure means the plain correlations survive that adjustment and KMO climbs toward 1; variables that only correlate in unrelated pairs push it down. Kaiser's labels run from marvelous above .90 through middling in the .70s to unacceptable below .50, and .60 is the usual working floor. A clean two-factor dataset scores around .91; pure noise scores about .50.

Bartlett's test of sphericity asks a much weaker question: could your correlation matrix have come from variables that are all mutually uncorrelated? Treat a significant result as a formality rather than an achievement. With 20 variables and 100 people, true correlations of just .10 already give p = .00005, and at 300 people the same tiny correlations give p below 10⁻¹⁶. A non-significant Bartlett means stop; a significant one means almost nothing on its own, so read KMO and the scree plot for the real answer.

Reading the result

Each component has loadings: how strongly each original variable contributes to it. You interpret a component by looking at which variables load heavily on it: if questions about parties, talking, and meeting strangers all load on one factor, you might name it "sociability." Naming factors is part science, part art.

How much of an item does the model explain?

A loading answers one question about one item and one factor. The follow-up is how much of that item the whole model accounts for, and that quantity has its own name. The communality, written h², is the sum of an item's squared loadings across the factors you kept. Its complement, 1 − h², is the item's uniqueness: variance belonging to that item alone, part genuinely specific content and part measurement error. An item loading .70 on a single factor has h² = .49, so the model explains just under half of it.

Fit a one-factor model to the eight wellbeing items in wellbeing-survey.csv (after reverse-coding q3 and q6) and the loadings land between .67 and .76, with communalities from .44 to .58 and a mean of .54. One factor is enough: χ²(20) = 20.14, p = .45. Those same eight items have a Cronbach's α of .90. Both numbers are correct, and reading them together is the useful part: any single item is about half noise, while their average is not, because the item-specific halves are unrelated to each other and largely cancel. That is the argument for scoring a scale instead of asking one good question, and it is reliability viewed from the item's side.

Communality is also where the two methods part company mechanically. PCA analyzes the correlation matrix with 1s down the diagonal, treating every variable as fully explainable and repackaging all of the variance, error included. Factor analysis puts estimated communalities on that diagonal instead, so it models only the variance the items share. With strong items the two solutions look almost identical; with weak ones PCA reports larger loadings and more variance explained than the data can support.

Two habits follow. Read the communality column before dropping an item for a low loading, because an item can load modestly on each of three correlated factors and still be well explained. And treat a communality near .20 as a verdict on the item rather than on the model: whatever that item is measuring, the factors you kept do not contain it. If the items came from a questionnaire, the fix usually lives back in how the item was worded, not in the extraction settings.