Chi-Square Tests

Everything so far has compared means of numeric data. But what about counts: how many people picked option A vs. B, recovered vs. didn't, voted left vs. right? The chi-square (χ²) test works with frequencies in categories, and its most common job is asking: are two categorical variables associated, or independent?

Observed vs. expected

The whole test rests on one comparison. You have the observed counts (what actually happened). Then you compute the expected counts: what you'd see if the two variables were completely unrelated (each cell's expected count is just its row total × column total ÷ grand total). The bigger the gap between observed and expected, the more evidence of a real association:

χ² = Σ (observed − expected)² / expected

The Σ (sigma) is an instruction to add: work the fraction out for one cell, then do it for every cell and total them.

That formula, the expected-count rule, the df and Cramér's V are four consecutive rows on the printable formula sheet, worth having beside you the first few times.

Work one cell, then the next, then add

The formula asks for a sum, and the way to get that sum right on paper is to postpone it. Take the cells in whatever order you like. For each one write down its expected count, subtract, square, divide. You end up with a small pile of numbers called the components, and χ² is their total.

Start with the trial in the builder below. Its first cell holds 45 recoveries where independence expects 65 × 70 ÷ 130 = 35.0, so that cell contributes (45 − 35.0)² ÷ 35.0 = 2.86. The other three come to 3.33, 2.86 and 3.33, and χ² = 12.38.

Two things make this worth the extra line of working. An arithmetic slip stays trapped in one cell instead of vanishing into the total with nothing to mark it, and the components are a finding in their own right: the largest ones mark the cells where the table has drifted furthest from independence, and those are the cells a results paragraph should name.

Reading the result

Degrees of freedom for a table are (rows − 1) × (columns − 1), which is 1 for the 2×2 trial and 6 for a three-by-four. A χ² far from zero, with p < 0.05, means counts like these would be surprising if the two variables were truly independent, so you conclude they are associated. In the trial: treatment and recovery are related.

The bars under the table say the same thing without any arithmetic. Each row's split across the columns is that row's conditional distribution, and the faint bar at the foot is the split of everybody together. Independence is exactly the claim that every row's conditional distribution matches that bottom bar, so χ² is measuring how far the rows have wandered from it. Rows that sit on top of one another give a χ² near zero however many cases the table holds.

On paper the p-value arrives as a bracket rather than a number. Take χ² and its df to Table F, find the df row, and walk along it until your statistic falls between two entries; the p-value lies between those two column headings. The builder prints that reading beside the exact one, so you can check a hand answer against the machine answer without leaving the page.

Here is a bigger one to try it on. Three neighborhoods by four ways of getting to work, 300 people in all: load it into the builder, and type the names in over the generic ones a loaded table arrives with, rows center, inner ring, outer ring and columns walk, bike, transit, car. χ² comes to 18.77 on 6 degrees of freedom. Table F's df 6 row carries 18.55 under .005 and 20.25 under .0025, so the paper answer is "between .005 and .0025" and the exact p is .0046. The tint says where it came from: walking in the center and driving in the outer ring account for more than half of χ² between them, while the middle row contributes almost nothing.

Where the curve comes from

χ² is not an invention for tables. Square a standard normal variable and you have a χ² distribution on one degree of freedom; add k independent squares and you have χ² on k. A contingency table's statistic is a sum of squared, standardized discrepancies, so the same family describes it. That parentage explains the rest of its behavior. χ² can never be negative, a table sitting exactly on its expected counts returns zero, and every p-value here is a right-tail area, because only large values are evidence against independence. The distribution playground holds the family among its nine curves, and moving the df slider shows the shape drifting right and turning symmetric as df grows.

Two flavors of chi-square

• Test of independence is the one above: are two categorical variables related? (Treatment × outcome, gender × preference.)
• Goodness-of-fit: does one categorical variable match an expected distribution? (Is this die fair? Do customers pick the four flavors equally?)

What you're assuming

Counts have to be independent: each observation falls in exactly one cell, and one person's category does not affect another's. Sixty people should produce sixty entries in the table.

The expected counts also have to be big enough for the χ² curve to describe the statistic well. Textbooks state that in three parts, and the builder checks all three live as you type:

• every expected count is at least 1;
• no more than 20% of the expected counts fall below 5;
• on a 2×2 table, where 20% of four cells rounds down to none of them, all four expected counts reach 5.

The rule is about expected counts, not observed ones, which is the part that catches people out. An observed zero is not by itself a problem. A cell that independence expected to hold two people is. When a table fails, the honest moves are to merge categories that genuinely belong together, collect more data, or on a 2×2 switch to Fisher's exact test, which computes the probability of the table directly and needs no approximation at all. Counts are cheap to collect and easy to under-collect, so if the study is still ahead of you, the power calculator will say how many observations a given association needs.

The 2×2 table you have already met

Cross a two-group variable with a yes/no outcome and you have exactly the data a two-proportion test uses. The two procedures are not merely similar, they are algebraically the same test, with χ² = z² on one degree of freedom and the same p-value to every decimal. Inference for proportions works that identity through on 270 people and says why the z version is usually the one to report: it hands you a signed difference and a confidence interval for it, where χ² hands you a positive number and nothing else. Past 2×2 the z test is no longer available, and χ² is the tool.

When the same person appears twice

Two different meanings of "independent" meet in this test, and only one of them is the null hypothesis. The test asks whether the two variables are independent. The assumption above is about the observations: sixty people should produce sixty entries, not a hundred and twenty.

Sixty students say whether they feel confident reading a results section, once before a workshop and once after. Stack the two occasions and you have a 2 × 2 table of 120 answers, which looks like a perfectly ordinary chi-square problem. It is not one. Each student sits in that table twice, so the table has 60 independent units, and a test built on 120 will describe a study nobody ran.

The fix is to make the person the row. Cross each student's before-answer with their own after-answer and every student lands in exactly one of four cells: yes both times, no both times, or one of the two switchers. The students who answered the same way twice tell you nothing about whether the workshop changed anything, so McNemar's test throws them out and works only on the switchers, which textbooks call the discordant pairs:

χ² = (b − c)² ⁄ (b + c), with 1 degree of freedom

where b and c are the two off-diagonal counts. Suppose 18 students went from no to yes and 6 went the other way. Then χ² = (18 − 6)² / 24 = 6.00 and p = .014. SPSS and R apply a continuity correction by default, giving 5.04 and p = .025; with only 24 switchers you can skip the approximation entirely and ask the binomial directly, which returns p = .023. All three say the same thing, and the exact one is the one to quote when b + c is small.

What does stacking the occasions actually cost? Simulating 60 students whose two answers correlate at .6, with no real change at all, the ordinary chi-square on the 120 rows rejects about 1% of the time rather than 5%. It is too cautious, not too eager, which catches most people out. Give the same students a genuine shift, from 60% confident to 78%, and McNemar finds it 78% of the time against the stacked test's 60%. Break the link between the two answers so that the pairing carries no information and the two tests land on top of each other, at 58% each. Pairing is worth using precisely when there is a pairing to use, and ignoring it here wastes participants rather than manufacturing findings.

Three or more occasions on the same people follow the same logic under a different name, Cochran's Q, which generalizes McNemar to k repeated yes/no measurements. The printable assumptions poster names McNemar as the fix for this row; the reason it is a fix, and what happens when the repetition is people inside schools rather than occasions inside people, is mixed & multilevel models.

Problem 21 in the practice problems runs a 2 × 3 table the paper-exam way, from its margins through six components to a bracket off Table F and the two row percentages that turn it into a sentence. Problem 19 takes a different 2 × 3 on to Cramér's V.