Non-Parametric Alternatives

When the assumptions behind a t-test or ANOVA don't hold (heavy skew, nasty outliers, tiny samples), you don't have to give up. Non-parametric tests throw away the raw numbers and work with their ranks instead. That one move makes them remarkably tough.

The trick: rank, don't measure

Most parametric tests care about how far apart values are. Rank-based tests only care about order: who sits above whom. Replace the numbers [2, 5, 9, 400] with their ranks [1, 2, 3, 4] and that monstrous 400 becomes just "the biggest one." Its ability to distort the result evaporates. That robustness to outliers and skew is the whole appeal.

The asymmetry is worth pausing on. As the highlighted value rockets upward, it inflates group 2's mean and its variance at the same time, yanking the t-statistic around. Sometimes the t-test even loses significance, because the variance blew up faster than the means pulled apart. The Mann-Whitney test barely reacts: the value was already the highest, so making it higher changes no ranks, and its p-value holds rock-steady.

Parametric, non-parametric, distribution-free

The vocabulary sounds forbidding and turns out to be simple. A parametric test names the distribution it believes the data came from and then estimates that distribution's parameters from the sample. The t-test assumes normal populations and estimates their means and standard deviations; those numbers are the parameters, and everything the test does is built on them. A non-parametric test names no distribution at all. Its older name, distribution-free, is the more honest of the two, because these tests still assume plenty (independent observations, and for several of them similarly shaped distributions in the groups being compared). What they give up is the requirement that you know the shape.

Ranking is what buys that freedom, and the mechanism is worth seeing. Whatever the population looked like, the ranks of N observations are always the whole numbers 1 to N. So the behavior of a rank statistic when nothing is going on can be worked out once, by counting arrangements, and reused for every dataset of that size forever. That is why these tests come with printed critical-value tables, and why the three this lesson reads are indexed by sample size alone rather than by degrees of freedom. Kruskal–Wallis and Friedman are the exception that shows the rule, since both hand their statistic to a χ² and so carry df of their own.

Each test then opens the same way, which is worth learning as one recipe rather than five procedures. Pool the values you are comparing, or reduce each matched pair to a single difference. Rank them from smallest to largest, giving tied values the average of the ranks they were competing for, so that two values fighting over ranks 4 and 5 both take 4.5 and the next value still takes 6. Sum the ranks belonging to each group. Then look up whatever statistic those sums produce. The table further down pairs each parametric test with the rank-based stand-in that takes over when its assumptions give way.

The logic behind each test

Every rank-based test starts from the same instinct: swap the numbers for their order, then ask a question the order can answer. What changes from test to test is which order you build, and what you add up once you have it.

Mann–Whitney U: counting wins

Pool both groups, sort everyone into a single line, and hand out ranks from lowest to highest. The statistic U counts wins: across every pairing of a group-1 value against a group-2 value, how often does one side come out on top? That count falls straight out of the rank sums, because a group's total rank tells you how high its members sit in the shared order. Divide the winning count by the number of pairings and you get the probability of superiority: the chance that a value drawn from one group beats a value drawn from the other. A lone extreme value can only ever be "the largest rank," so pushing it further out changes nothing.

On paper nobody counts those wins one pairing at a time. Rank the pooled values, add up the ranks belonging to group 1 to get R₁, and the count drops out of a formula:

U₁ = n₁n₂ + n₁(n₁ + 1) / 2 − R₁

Swap every subscript for U₂. The two are locked together by U₁ + U₂ = n₁n₂, the total number of cross-group pairings, so working the second one out from R₂ instead of by subtraction costs nothing and checks your ranking for free. The test statistic is the smaller of the two, and its table points the opposite way to the ones you are used to: reject the null when U is at or below the critical value.

The ledger below runs on six control values and six treatment values, and its rank sums are R₁ = 23 and R₂ = 55. That gives U₁ = 36 + 21 − 23 = 34 and U₂ = 36 + 21 − 55 = 2, which add to 36 as they must, so the statistic is 2. Table U at n₁ = n₂ = 6 and two-tailed α = .05 gives U* = 5, and 2 sits comfortably under it, so the null goes. The exact p of .009 is the same verdict reached from the other end.

Some textbooks define U₁ the other way round, as R₁ − n₁(n₁ + 1) / 2. That version returns the number this one calls U₂, so you finish with the same pair either way and the smaller of them is the same number. Nothing in the decision depends on which convention your course prints.

Wilcoxon signed-rank: magnitudes with a sign

For paired data, work with each person's difference score. Rank the absolute differences from smallest to largest, then re-attach the original signs. Adding up the ranks that carried a plus sign gives W₊ (the minus ranks give W₋, and the two are locked together). A plain sign test would only count how many people improved; the signed-rank test also weights how much each one moved, so a big change pulls harder than a barely-there one. The magnitudes enter only through their rank order, which is why the person with the wildest swing contributes the top rank whether they gained 13 points or 130.

The hand procedure has one step people forget. Take each pair's difference, then throw away every pair whose difference is exactly zero and let n be the number of pairs still standing, because someone who did not move tells you nothing about direction and inventing a rank for them would be a fiction. Rank what is left by absolute size. Add the ranks that carried a plus sign to get T₊ and the ranks that carried a minus sign to get T₋; those two have to add to n(n + 1) / 2, which is the check. W is the smaller of them, and this table is a ceiling too: reject when W is at or below the critical value.

The Wilcoxon tab below follows seven people whose differences are 7, 5, −1, 11, 8, 13 and 9. Ranking those by size gives 3, 2, 1, 6, 4, 7 and 5, so T₊ = 27 and T₋ = 1, and 27 + 1 is 28, which is 7 × 8 / 2 as promised. W = 1 against Table W's W* = 2 at n = 7 and two-tailed α = .05, so the null goes here as well, exact p = .031.

Kruskal–Wallis: ANOVA on the ranks

Three or more independent groups reuse the same opening move: pool everyone, rank the whole pile, and total the ranks inside each group. If the groups were interchangeable, those rank totals should land roughly proportional to group size. The statistic H measures how far the observed totals stray from that even split, and under the null hypothesis it follows a chi-square distribution with k − 1 degrees of freedom. It is one-way ANOVA with ranks standing in for the raw scores.

Friedman: ranking within each person

Repeated measures rank within each person rather than across the whole sample. For every participant, rank their conditions from lowest to highest, so each row of the data turns into a small 1-2-3. Add those ranks down each column: if one condition genuinely tends to score highest, its column keeps collecting the top ranks. The statistic reads that imbalance and, like Kruskal–Wallis, compares it to a chi-square distribution with k − 1 degrees of freedom. Ranking inside each person is what cancels stable individual differences, the same trick the paired difference pulls for Wilcoxon.

Spearman's ρ_s: correlation on the ranks

Spearman's ρ_s (rho, with a subscript s so nobody confuses it with the population correlation) is Pearson's r computed on ranks instead of values. Rank the x column, rank the y column separately, hand those two columns of ranks to the ordinary correlation formula, and what comes back measures whether the relationship climbs or falls steadily rather than whether it follows a straight line. An outlier can only ever be the top rank, so it cannot drag the coefficient the way it drags r.

By hand nobody runs the correlation formula. Take each pair's rank difference d, square it, add the squares, and use the shortcut:

ρ_s = 1 − 6Σd² / (n(n² − 1))

Two things about that shortcut deserve knowing before you trust it. It is algebraically exact only when nothing ties, because it is derived on the assumption that both rank columns are the whole numbers 1 to n, which stops being true the moment an average rank appears. With ties, go back to the definition and run Pearson on the two rank columns, which is what §1.17 already says Spearman is and what your software does either way. The sign is also doing real work: Σd² of zero means the two rankings agree perfectly and ρ_s = 1, while the largest Σd² a design can produce comes from exactly reversed rankings and returns −1.

ρ_s then gets a table of its own, and this one is a floor again: reject when |ρ_s| reaches the critical value. The Spearman tab below ranks ten students on minutes spent with the practice set and score on the follow-up test. Σd² comes to 50, so ρ_s = 1 − 300 / 990 = .697, and Table ρ at n = 10 and two-tailed α = .05 asks for .648. It clears, though not by a wide margin, and the exact p of .031 tells the same story.

That tab also carries the lesson's opening argument in miniature. Drag the ringed student's score upward and watch the two coefficients part company: Pearson on the raw values slides from .71 down toward .62 as the point pulls away from everyone else, while Σd², ρ_s and the verdict do not move at all. The student was already ranked highest, and being further ahead cannot make them any more highest than that.

Reading a rank table

Two of these tables point the opposite way to every other table in the course, and that is what catches people out under time pressure. U counts overlap and W counts the ranks stacked on the losing side, so both get smaller as the groups pull apart. Their tables print ceilings, and the significant statistic is the one that falls to the cell or below. Spearman's ρ_s is a correlation and behaves like one, so its table is a floor you have to reach. All three live on the tables page, generated by counting rank orders rather than copied out of a book, next to a lookup drill that mixes them in with z, t and χ².

Most printed rank tables give you a two-tailed column and a one-tailed column side by side. When yours carries only one of them the conversion is arithmetic rather than a different table, because a two-tailed test spends half its α in each tail: a one-tailed test at α reads the same cell as a two-tailed test at 2α. So a one-tailed test at .05 reads the two-tailed .10 column, and a two-tailed .05 column is already the one-tailed .025 cutoff. Decide the direction before you look, never after.

One last thing about the cells themselves. A rank statistic can only take a finite set of values, so no cell has a tail area of exactly .05. The table keeps the last value whose tail is still at or under α, which makes every one of these tests slightly stricter than its label claims. With seven pairs, Wilcoxon's two-tailed .05 cell of W* = 2 actually carries a probability of .047, and the next value up carries .078. There is nothing in between, and no amount of arithmetic will produce it.

That conservatism is part of the price. On data the t-test was built for, Mann–Whitney needs about 105 people to learn what a t-test learns from 100, an efficiency of roughly 95%, and the paired versions sit in the same neighborhood. It is a small standing charge, and what it buys is an answer that does not come apart when one value is absurd.

Which test replaces which

Each rank-based test steps in for a specific parametric one when its assumptions give way. The test chooser walks the same map from the other direction, starting from your design rather than from the test you were hoping to run:

After a significant H

Kruskal–Wallis inherits one more thing from ANOVA: it is an omnibus test. A small p says the k groups are not interchangeable and stops there. The rank-based follow-up is Dunn's test, which compares two groups' mean ranks using the ranking the omnibus already computed. With no ties, the standard error for a pair is √[N(N + 1)/12 × (1/n_i + 1/n_j)], the difference in mean ranks divided by it is treated as a z, and the resulting p-values get the same family-wise correction you would apply after an ANOVA.

The temptation is to skip all that and run three Mann–Whitney tests instead. Fifteen scores show why that is a different analysis rather than a shortcut. Take the numbers 1 to 15, so every value is its own rank, split across three groups:

Kruskal–Wallis gives H = 10.5 on 2 df, p = .005. Dunn's test then puts A against B at z = 3.18 and A against C at z = 1.06, and B against C at z = 2.12, which after a Holm correction leaves only A versus B below .05 (.004, against .068 for B versus C). Run the three Mann–Whitney tests instead, correct them the same way, and B versus C comes back significant at .024.

Both answers are arithmetically right, because they rank different things. Every B score beats every C score, so a test that sees only those ten values finds the most extreme separation five-on-five can produce. Dunn sees all fifteen, and in the full ladder B and C sit next to each other with six mean-rank points between them. Only one of those two views matches the H you just reported, which is the argument for Dunn: a pairwise test discards the third group and re-ranks, so its ranks are not the ranks the omnibus used. Friedman needs the same care for the same reason, and its follow-up ranks within each participant rather than pooling.

The effect size your software prints

A rank test still owes you a size, not just a verdict. The parametric side has Cohen's d and η²; the rank side has the rank-biserial correlation, and the ledger above is already showing it to you under another name.

Recall the probability of superiority: the chance that a randomly picked value from one group beats a randomly picked value from the other. The rank-biserial correlation is that same quantity restretched onto the −1 to +1 scale a correlation lives on:

r_rb = 2 × P(superiority) − 1

Equivalently, it is the win rate minus the loss rate across every cross-group pairing. On the Mann–Whitney tab's starting data the ledger reports P(superiority) = .94, which puts r_rb at .89. Read it as a scoreboard: of the 36 possible pairings of one control value against one treatment value, treatment wins 34 and loses 2, and .89 is the gap between those two rates. Zero means the two groups interleave evenly; a magnitude of 1 means every value in one group beats every value in the other, and the sign records which group that is. (The ledger always reports the winning side's probability of superiority, so the formula above hands you the magnitude.)

Wilcoxon has a matched-pairs version of the same idea, (W₊ − W₋) / (W₊ + W₋), built from the signed-rank sums the ledger already displays.

Where this bites in practice: JASP prints the rank-biserial correlation when you tick Effect size in the analysis panel, and SPSS does not, which is why so many write-ups fall back on r = z/√N instead. Those two are different statistics, and on this same data they disagree: z = 2.48 over √12 gives r = .72 against the rank-biserial .89. Both are defensible; reporting which one you computed is not optional. If JASP is new to you, the thesis-data walkthrough takes a raw file through to a finished APA sentence.

The trade-off

Problem 18 of the practice problems runs a Mann–Whitney by hand on five dogs per shelter, from ranking the ten values through to the decision at α = .05. Problem 27 takes the same test to a table and misses the cell by one, Problem 28 works a Wilcoxon with a zero difference in it, and Problem 29 lands a Spearman ρ_s exactly on its critical value.