Independent Samples t-Test

This is the test you will use most often: do two separate groups (treatment vs. control, men vs. women, version A vs. version B) differ on average? Unlike the paired test, the groups are made of different individuals, so we compare their two means directly.

Signal over noise

The logic is a ratio. On top, the signal: how far apart the two sample means are. On the bottom, the noise: how much the scores bounce around within each group, adjusted for sample size. That ratio is the t-statistic:

t = (x̄₁ − x̄₂) / (standard error of the difference)

A big t means the gap stands out clearly from the noise, so the groups probably differ. A small t means the gap could easily be chance. The same mean difference can be convincing or meaningless, depending on the noise and the sample size. The playground below shows this.

Written out, that standard error is assembled from each group's own spread and its own sample size:

SE = √( s₁²/n₁ + s₂²/n₂ )

Each group contributes its variance divided by its sample size. The two contributions are added, and the square root is taken at the end. You never add the two standard errors themselves. That looks like a small detail, but it causes the most common mistake in this chapter, and the second interactive on this page is about it.

Three things that change t

• Bigger gap → bigger t. A larger real difference is easier to detect.
• More noise → smaller t. If scores within each group are all over the place, a gap between the means is less convincing.
• Bigger samples → bigger t. More data per group shrinks the standard error, so even a modest gap can become significant.

Try this: set a gap of 8 with high noise and small n, which is likely not significant. Now slide n up. The means barely move, but the p-value drops below 0.05. The effect is the same and the evidence is stronger. This is why sample size and power matter so much.

Three degrees of freedom for one test

The formula above fixes the standard error. It does not fix which t-distribution to compare the answer with, and there are three defensible choices. Each one is a way of deciding how much to trust the estimate of the noise.

• The conservative hand rule takes df = min(n₁ − 1, n₂ − 1). It is the smaller group's degrees of freedom, a deliberately pessimistic choice, and exams expect it because you can work it out without a calculator.
• Welch–Satterthwaite combines the two variances and two sample sizes into a fractional df, usually well above the conservative one. Statistics software computes it for you. The reasoning behind it is in the Welch's ANOVA section, which generalizes it.
• The pooled rule takes df = n₁ + n₂ − 2, the largest of the three. It has a condition: the two populations must have the same variance, so that both samples can be combined into a single estimate of it. The standard error is then rebuilt from that pooled estimate as well.

Suppose ten students taught one way average 78.0 with s = 9.0, and sixteen taught another way average 70.0 with s = 7.0. The unpooled standard error is √(81/10 + 49/16) = √11.163 = 3.341, so t = 8.0/3.341 = 2.394.

The first two rows share a standard error and a t, and differ only in the t-distribution used to read them. The third row changes the statistic too, because pooling rebuilds the standard error from one combined variance: s_p = 7.810 gives SE = 3.148 and t = 2.541.

The p column shows why the hand rule is safe. Fewer degrees of freedom mean heavier tails, a larger critical value, a wider interval and a larger p-value, so the conservative rule can only make you more cautious than the software. If your test clears α with df = 9, it clears it with df = 15.74 as well. So an exam accepts the conservative answer without qualification: it can miss a significant result but never invent one. When you use software, report its df, and expect it to differ from the one you would write by hand.

The interval for the difference

The same numbers also give an interval, built the same way as the one-sample interval: keep the estimate and the standard error, and use the critical value to set the margin.

(x̄₁ − x̄₂) ± t* · SE

The interval is for the difference, so a zero inside it says the data are compatible with no difference at all. A non-significant test says the same. In the worked numbers above, the conservative interval [0.44, 15.56] misses zero by a little and the pooled interval [1.50, 14.50] misses it by more, matching the p column. The width matters as much as the decision. An interval from half a point to fifteen and a half does not show a big effect. It shows that the study was too small to say how big the effect is.

One simpler test appears only in textbooks. If σ₁ and σ₂ were both known, nothing in the denominator would be estimated. The SE would be exactly √(σ₁²/n₁ + σ₂²/n₂), the critical value would come from the normal curve, and the df question would disappear. That is the two-sample z, and real data almost never come with the two σ values it needs.

Two intervals that overlap, one difference that does not

Journal figures often show each group's mean with its own 95% interval, and readers often use a shortcut: if the bars overlap, the groups do not differ. The shortcut is wrong often enough to matter, because of how the standard error of a difference is built.

At the default gap of 8 points the two intervals overlap by 3.23 points, so the shortcut says the groups do not differ. The test says they do: t(38) = 2.11, p = .042, and the interval for the difference, [0.32, 15.68], excludes zero. Both pictures come from the same twenty-plus-twenty numbers, so one of them is being misread, and it is the top one.

The arithmetic shows why. Each group's margin here is 2.093 × 2.683 = 5.62, so the two margins together span 11.23 points, and the bars keep touching until the means are that far apart. The margin on the difference is 2.024 × 3.795 = 7.68, and 3.795 is not 2.683 + 2.683. Standard errors combine in quadrature, as √(SE₁² + SE₂²), because the two errors are independent. Sometimes one group's mean runs high while the other runs low, which exaggerates the gap. About as often, both run the same way and the gap hardly changes. Adding the two SEs directly would be right only if every high in one group came with a low in the other, and independence rules that out.

For an exam, remember the rule in one direction only. Intervals that do not overlap do mean a significant difference, at least when the two groups have similar sizes and spreads, as in the figure above and nearly every one you will meet. Intervals that do overlap tell you nothing on their own. The range where they overlap while the test rejects is wide: on these numbers it runs from a gap of 7.7 points to a gap of 11.2. So if a figure shows per-group intervals and no test, and the bars touch, you cannot tell whether the groups differ.

What you're assuming

• The two groups are independent (different people, no pairing).
• Each group is roughly normal, or both samples are large enough for the CLT. When neither is true, the Mann-Whitney test asks a closely related question without needing normality at all.
• Similar spread in both groups. If not, the Welch version of the test relaxes that requirement.

The same test on real data: sleep-experiment.csv gives 80 volunteers split across a normal and a restricted night, with t, p and d worked out for you to check against.

Two practice problems run on this lesson. Problem 12 runs the test on twenty-four children, then asks you to judge a researcher who read a non-significant result as proof that the two methods work equally well. Problem 20 works the same test twice, once with the conservative degrees of freedom and once with Welch's.