Non-Parametric Alternatives

When the assumptions behind a t-test or ANOVA don't hold (heavy skew, nasty outliers, tiny samples), you don't have to give up. Non-parametric tests throw away the raw numbers and work with their ranks instead.

The trick: rank, don't measure

Rank-based tests resist outliers and skew because they use only the order of the values. Most parametric tests also use how far apart the values are. Replace the numbers [2, 5, 9, 400] with their ranks [1, 2, 3, 4] and the 400 becomes simply the largest value, so it can no longer distort the result.

As the highlighted value moves upward, it raises group 2's mean and its variance at the same time, so the t-statistic swings. Sometimes the t-test even loses significance, because the variance grew faster than the gap between the means. The Mann-Whitney test barely reacts. The value was already the highest, so making it higher changes no ranks, and the p-value stays the same.

Parametric, non-parametric, distribution-free

A parametric test assumes a particular distribution for the data and estimates that distribution's parameters from the sample. The t-test assumes normal populations and estimates their means and standard deviations. Those are the parameters, and the whole test is built on them. A non-parametric test assumes no particular distribution. Its older name, distribution-free, describes these tests better, because they still make assumptions (independent observations, and for several of them similarly shaped distributions in the groups being compared). They just do not require you to know the shape.

Ranking makes this possible. Whatever the population looked like, the ranks of N observations are always the whole numbers 1 to N. So the behavior of a rank statistic under the null hypothesis can be worked out once, by counting arrangements, and reused for every dataset of that size. That is why these tests come with printed critical-value tables, and why the three tables this lesson reads are indexed by sample size alone, with no degrees of freedom. Kruskal–Wallis and Friedman are the exception: both compare their statistic to a χ² distribution, so they have df of their own.

Each test starts the same way, so you can learn the steps once. Pool the values you are comparing, or reduce each matched pair to a single difference. Rank them from smallest to largest. Tied values get the average of the ranks they share, so two values tied for ranks 4 and 5 both get 4.5, and the next value still gets 6. Sum the ranks belonging to each group. Then look up whatever statistic those sums produce. The table further down pairs each parametric test with the rank-based test that replaces it when its assumptions fail.

The logic behind each test

The tests differ in which values get ranked together and in what is added up afterwards.

Mann–Whitney U: counting wins

Pool both groups, sort everyone into a single line, and assign ranks from lowest to highest. The statistic U counts wins: across every pairing of a group-1 value with a group-2 value, how often does one side come out on top? You can get that count from the rank sums, because a group's total rank tells you how high its members are in the shared order. Divide the winning count by the number of pairings and you get the probability of superiority: the chance that a value drawn from one group beats a value drawn from the other. A lone extreme value can only ever have the largest rank, so pushing it further out changes nothing.

By hand, you do not count those wins one pairing at a time. Rank the pooled values, add up the ranks belonging to group 1 to get R₁, and use this formula:

U₁ = n₁n₂ + n₁(n₁ + 1) / 2 − R₁

For U₂, swap every subscript. The two always satisfy U₁ + U₂ = n₁n₂, the total number of cross-group pairings. So compute U₂ from R₂ as well: if the two do not add to n₁n₂, there is a mistake in your ranking. The test statistic is the smaller of the two. Its table works the opposite way to the ones you are used to: reject the null when U is at or below the critical value.

The ledger below uses six control values and six treatment values, with rank sums R₁ = 23 and R₂ = 55. That gives U₁ = 36 + 21 − 23 = 34 and U₂ = 36 + 21 − 55 = 2. They add to 36 as they should, and the statistic is the smaller one, 2. Table U at n₁ = n₂ = 6 and two-tailed α = .05 gives U* = 5. Since 2 is well below 5, reject the null. The exact p of .009 gives the same decision.

Some textbooks define U₁ the other way round, as R₁ − n₁(n₁ + 1) / 2. That version returns the number this one calls U₂, so you finish with the same pair either way and the smaller of them is the same number. Nothing in the decision depends on which convention your course prints.

Wilcoxon signed-rank: magnitudes with a sign

For paired data, work with each person's difference score. Rank the absolute differences from smallest to largest, then re-attach the original signs. Adding up the ranks that carried a plus sign gives W₊ (the minus ranks give W₋, and the two add to a fixed total). A plain sign test would only count how many people improved. The signed-rank test also takes into account how much each one moved, so a big change counts for more than a small one. But the sizes enter only through their rank order. The person with the largest change gets the top rank whether they gained 13 points or 130.

The hand procedure has one step people forget. Take each pair's difference, then throw away every pair whose difference is exactly zero, and let n be the number of pairs left. Someone who did not change tells you nothing about direction, so they get no rank. Rank what is left by absolute size. Add the ranks that carried a plus sign to get T₊ and the ranks that carried a minus sign to get T₋. As a check, the two must add to n(n + 1) / 2. W is the smaller of them, and this table is a ceiling too: reject when W is at or below the critical value.

The Wilcoxon tab below follows seven people whose differences are 7, 5, −1, 11, 8, 13 and 9. Ranking those by size gives 3, 2, 1, 6, 4, 7 and 5, so T₊ = 27 and T₋ = 1. The check works: 27 + 1 = 28 = 7 × 8 / 2. W = 1 is below Table W's W* = 2 at n = 7 and two-tailed α = .05, so the null is rejected here as well (exact p = .031).

Kruskal–Wallis: ANOVA on the ranks

With three or more independent groups, the first steps are the same: pool everyone, rank the whole set, and total the ranks inside each group. If the groups were interchangeable, those rank totals should be roughly proportional to group size. The statistic H measures how far the observed totals are from that even split. Under the null hypothesis it follows a chi-square distribution with k − 1 degrees of freedom. It is one-way ANOVA with ranks in place of the raw scores.

Friedman: ranking within each person

With repeated measures, the ranking happens within each person. For every participant, rank their conditions from lowest to highest, so each row of the data becomes 1, 2, 3. Add those ranks down each column. If one condition tends to score highest, its column collects most of the top ranks. The statistic measures that imbalance and, like Kruskal–Wallis, compares it to a chi-square distribution with k − 1 degrees of freedom. Because each person is ranked only against themselves, stable differences between people cancel out, just as they do in the paired differences Wilcoxon uses.

Spearman's ρ_s: correlation on the ranks

Spearman's ρ_s (rho, with a subscript s to tell it apart from the population correlation) is Pearson's r computed on ranks. Rank the x column, rank the y column separately, and put the two columns of ranks into the ordinary correlation formula. The result measures whether one variable consistently rises (or falls) as the other rises, even if the relationship is not a straight line. An outlier can only ever take the top rank, so it cannot pull the coefficient the way it pulls r.

By hand, use a shortcut in place of the correlation formula. Take each pair's rank difference d, square it, and add the squares:

ρ_s = 1 − 6Σd² / (n(n² − 1))

Two facts about the shortcut matter. First, it is algebraically exact only when nothing ties. Its derivation assumes both rank columns are the whole numbers 1 to n, and that stops being true as soon as an average rank appears. With ties, go back to the definition and run Pearson on the two rank columns. That matches the definition of Spearman in §1.17, and your software does it either way. Second, the formula runs from 1 down to −1. Σd² of zero means the two rankings agree perfectly and ρ_s = 1, while the largest Σd² possible with n pairs comes from exactly reversed rankings and gives −1.

ρ_s has a table of its own, and this one works the usual way: reject when |ρ_s| reaches the critical value. The Spearman tab below ranks ten students on minutes spent with the practice set and score on the follow-up test. Σd² comes to 50, so ρ_s = 1 − 300 / 990 = .697. Table ρ at n = 10 and two-tailed α = .05 requires .648, so ρ_s is significant, though not by a wide margin. The exact p of .031 agrees.

That tab also repeats the lesson's opening demonstration. Drag the ringed student's score upward and the two coefficients separate. Pearson on the raw values falls from .71 toward .62 as the point moves away from everyone else, while Σd², ρ_s and the verdict do not change at all. The student was already ranked highest, and a higher score cannot raise that rank.

Reading a rank table

Two of these tables work the opposite way to every other table in the course, and students often get this wrong under time pressure. U counts overlap and W counts the ranks on the losing side, so both get smaller as the groups move apart. Their tables print ceilings: the statistic is significant when it falls to the cell value or below. Spearman's ρ_s is a correlation and behaves like one, so its table is a floor you have to reach. All three are on the tables page, next to a lookup drill that mixes them in with z, t and χ².

Most printed rank tables give a two-tailed column and a one-tailed column side by side. If yours has only one, you can convert. A two-tailed test puts half its α in each tail, so a one-tailed test at α uses the same cell as a two-tailed test at 2α. A one-tailed test at .05 therefore reads the two-tailed .10 column, and the two-tailed .05 column is the one-tailed .025 cutoff. Decide the direction before you look, never after.

The cells have one more feature. A rank statistic can take only a finite set of values, so no cell has a tail area of exactly .05. The table keeps the last value whose tail is still at or under α, so each of these tests is slightly stricter than its stated α. With seven pairs, Wilcoxon's two-tailed .05 cell, W* = 2, has a tail probability of .047, and the next value up has .078. No value of W has a tail probability between those two.

That conservatism is one cost of using ranks. Another is a small loss of power. On normal data, the kind the t-test was designed for, Mann–Whitney needs about 105 people to match the power a t-test has with 100, an efficiency of roughly 95%. The paired versions are close to that figure. In exchange, the answer does not fall apart when one value is absurd.

Which test replaces which

Each rank-based test replaces a specific parametric one when that test's assumptions fail. The test chooser covers the same ground from the other direction, starting from your design:

After a significant H

Like ANOVA, Kruskal–Wallis is an omnibus test. A small p says the k groups are not interchangeable, and nothing more. The rank-based follow-up is Dunn's test, which compares two groups' mean ranks using the ranking from the omnibus test. With no ties, the standard error for a pair is √[N(N + 1)/12 × (1/n_i + 1/n_j)]. The difference in mean ranks divided by it is treated as a z, and the resulting p-values get the same family-wise correction you would apply after an ANOVA.

It is tempting to run three Mann–Whitney tests instead. That is a different analysis, not a shortcut, as fifteen scores show. Take the numbers 1 to 15, so every value is its own rank, and split them across three groups:

Kruskal–Wallis gives H = 10.5 on 2 df, p = .005. Dunn's test then gives z = 3.18 for A against B, 1.06 for A against C and 2.12 for B against C. After a Holm correction only A versus B is below .05 (.004, against .068 for B versus C). Run the three Mann–Whitney tests instead, correct them the same way, and B versus C comes out significant at .024.

Both answers are arithmetically correct, because they rank different things. Every B score beats every C score, so a test that uses only those ten values finds the largest separation possible with five against five. Dunn's test uses all fifteen, and in the full ranking B and C are neighbors with six mean-rank points between them. Dunn's test uses the same ranks as the H you just reported. A pairwise Mann–Whitney test drops the third group and ranks again, so its ranks are not the ones the omnibus test used. That is the argument for Dunn. Friedman needs the same care for the same reason, and its follow-up ranks within each participant rather than pooling.

The effect size your software prints

A rank test should be reported with an effect size too. Parametric tests have Cohen's d and η². Rank tests have the rank-biserial correlation, and the ledger above already shows it under another name.

Recall the probability of superiority: the chance that a randomly picked value from one group beats a randomly picked value from the other. The rank-biserial correlation is that same quantity rescaled to run from −1 to +1, like a correlation:

r_rb = 2 × P(superiority) − 1

Equivalently, it is the win rate minus the loss rate across every cross-group pairing. On the Mann–Whitney tab's starting data the ledger reports P(superiority) = .94, so r_rb = .89. Of the 36 possible pairings of one control value with one treatment value, treatment wins 34 and loses 2, and .89 is the difference between those two rates. Zero means the two groups interleave evenly. A magnitude of 1 means every value in one group beats every value in the other, and the sign records which group that is. (The ledger always reports the winning side's probability of superiority, so the formula above gives the magnitude.)

Wilcoxon has a matched-pairs version of the same idea, (W₊ − W₋) / (W₊ + W₋), built from the signed-rank sums the ledger already displays.

JASP prints the rank-biserial correlation when you tick Effect size in the analysis panel, and SPSS does not. As a result, many write-ups fall back on r = z/√N. These are different statistics, and on this data they disagree: z = 2.48 divided by √12 gives r = .72, against a rank-biserial .89. Both are defensible, but you must report which one you computed. If JASP is new to you, the thesis-data walkthrough takes a raw file through to a finished APA sentence.

The trade-off

Problem 39 of the practice problems runs a Mann–Whitney by hand on five dogs per shelter, from ranking the ten values through to the decision at α = .05. Problem 48 takes the same test to a table and misses the cell by one. Problem 49 works a Wilcoxon with a zero difference in it, and in Problem 50 a Spearman ρ_s equals its critical value exactly.