Independent Samples t-Test
This is the test you reach for constantly: do two separate groups (treatment vs. control, men vs. women, version A vs. version B) differ on average? Unlike the paired test, the groups are made of different individuals, so we compare their two means head to head.
Signal over noise
The logic is a ratio. On top, the signal: how far apart the two sample means are. On the bottom, the noise: how much the scores bounce around within each group, adjusted for sample size. That ratio is the t-statistic:
t = (x̄₁ − x̄₂) / (standard error of the difference)
A big t, a clear signal sticking out of the noise, means the groups probably really differ. A small t means the gap could easily be chance. Crucially, the same mean difference can be convincing or meaningless depending on the noise and the sample size. The playground makes that tug-of-war visible.
Written out, that standard error is assembled from each group's own spread and its own sample size:
SE = √( s₁²/n₁ + s₂²/n₂ )
Each group hands over its variance divided by its sample size, the two contributions add, and the square root is taken at the end. What never happens is adding the two standard errors themselves. That distinction looks like bookkeeping and it is the source of the most common mistake in this chapter, which the second interactive on this page is built around.
🎮 Two-Group Tug-of-War
Each dot is a person, and the same people stay on screen while you drag the sliders, so you can watch the groups slide apart or spread out. Hit “New samples” for a fresh random draw.
Three levers, one conclusion
- Bigger gap → bigger t. A larger real difference is easier to detect.
- More noise → smaller t. If scores within each group are all over the place, a gap between the means is less convincing.
- Bigger samples → bigger t. More data per group shrinks the standard error, so even a modest gap can become significant.
Try this: set a gap of 8 with high noise and small n: likely not significant. Now slide n up. The means barely move, but the p-value tumbles past 0.05. Same effect, more evidence. This is exactly why sample size and power matter so much.
Three degrees of freedom for one test
The standard error is settled by the formula above. What is not settled is which t-distribution to hold the answer against, and there are three defensible replies. They are not three tests; they are three ways of deciding how much the estimate of the noise should be trusted.
- The conservative hand rule takes df = min(n₁ − 1, n₂ − 1). It is the smaller group's degrees of freedom, which is deliberately pessimistic, and it is the rule an exam expects because you can write it down without a calculator.
- Welch–Satterthwaite works the two variances and two sample sizes into a fractional df that usually lands well above the conservative one. Statistics software computes it for you, and the reasoning behind it belongs with the Welch's ANOVA section that generalizes it, so it is not repeated here.
- The pooled rule takes df = n₁ + n₂ − 2, which is the largest of the three, and it comes with a condition: the two populations must share one variance, so that both samples can be combined into a single estimate of it. Then the standard error is rebuilt from that pooled estimate as well.
Numbers make the differences concrete. Ten students taught one way average 78.0 with s = 9.0; sixteen taught another way average 70.0 with s = 7.0. The unpooled standard error is √(81/10 + 49/16) = √11.163 = 3.341, so t = 8.0/3.341 = 2.394.
| Rule for df | df here | Critical t* | t | p | 95% CI for the difference |
|---|---|---|---|---|---|
| Conservative, min(n₁ − 1, n₂ − 1) | 9 | 2.262 | 2.394 | .040 | [0.44, 15.56] |
| Welch–Satterthwaite | 15.74 | 2.123 | 2.394 | .030 | [0.91, 15.09] |
| Pooled, n₁ + n₂ − 2 | 24 | 2.064 | 2.541 | .018 | [1.50, 14.50] |
The first two rows share a standard error and a t, and differ only in the curve those numbers are read against. The third row changes the statistic too, because pooling rebuilds the standard error from one combined variance: sp = 7.810 gives SE = 3.148 and t = 2.541.
Read down the p column and the pattern is the whole argument for the hand rule. Fewer degrees of freedom mean fatter tails, a larger critical value, a wider interval and a larger p-value, so the conservative rule can only ever make you more cautious than the software would be. If your test clears α with df = 9, it clears it with df = 15.74 as well. An exam therefore accepts the conservative answer without qualification: it cannot manufacture a significant result, only forgo one. Report the software's df when you have software, and expect it not to match the number you would have written by hand.
The interval for the difference
Everything above also builds an interval, and it is the same move the one-sample interval made: keep the estimate, keep the standard error, and put the critical value on the other side.
(x̄₁ − x̄₂) ± t* · SE
The estimate being bracketed is the difference, so a zero inside the interval says the data are compatible with no difference at all, which is the same message a non-significant test sends. In the worked numbers above, the conservative interval [0.44, 15.56] misses zero by a little and the pooled interval [1.50, 14.50] misses it by more, which is the p column saying the same thing in other units. The width is worth as much as the verdict: an interval running from half a point to fifteen and a half is not evidence of a big effect, it is evidence that the study was too small to say how big the effect is.
One rung below all of this sits a test you will meet only in a textbook. If σ₁ and σ₂ were both known, nothing would be estimated in the denominator, the SE would be √(σ₁²/n₁ + σ₂²/n₂) exactly, the critical value would come off the normal curve, and the whole df question would disappear; that is the two-sample z, and real data almost never hands you the two σ values it needs.
Two intervals that overlap, one difference that does not
Journal figures often show each group's mean with its own 95% interval, and readers pick up a shortcut from them: if the bars overlap, the groups do not differ. The shortcut is wrong often enough to matter, and it is wrong for a reason that follows directly from how the standard error of a difference is built.
🎮 Overlapping Intervals, Different Means
Twenty people per group, each group's own 95% interval drawn around its mean, and underneath them the interval for the difference between the two. Slide the groups apart and watch the moment the bottom interval clears zero, which is not the moment the two top intervals stop touching.
At the default gap of 8 points the two intervals overlap by 3.23 points, and a reader applying the shortcut would stop there. The test does not: t(38) = 2.11, p = .042, and the interval for the difference runs [0.32, 15.68], nowhere near zero. Both pictures are drawn from the same twenty-plus-twenty numbers, so one of them is being read wrongly, and it is the top one.
The arithmetic says why. Each group's margin here is 2.093 × 2.683 = 5.62, so laid end to end the two margins span 11.23 points, and the bars keep touching until the means are that far apart. The margin on the difference is 2.024 × 3.795 = 7.68, and 3.795 is not 2.683 + 2.683. Standard errors combine in quadrature, as √(SE₁² + SE₂²), because the two errors are independent: about as often as one group's mean runs high while the other runs low and the gap is exaggerated, both run the same way together and the gap is barely touched. Adding the two outright would be right only if every high in one group came paired with a low in the other, which is the one pattern independence rules out.
The asymmetry is worth remembering in the form an exam can use. Intervals that do not overlap do mean a significant difference, at least whenever the two groups are of comparable size and spread, which covers the figure above and nearly every one you will meet. Intervals that do overlap mean nothing on their own, and the range where they overlap while the test rejects is wide: on these numbers it runs from a gap of 7.7 points to a gap of 11.2. So if a figure hands you per-group intervals and no test, and the bars touch, the honest reading is that you cannot tell.
Report the effect size too. A significant t with a huge sample might reflect a difference too small to care about. Always pair the p-value with Cohen's d (the gap measured in standard deviations) to judge whether the difference actually matters.
What you're assuming
- The two groups are independent (different people, no pairing).
- Each group is roughly normal, or both samples are large enough for the CLT. When neither is true, the Mann-Whitney test asks a closely related question without needing normality at all.
- Similar spread in both groups. If not, the Welch version of the test relaxes that requirement.
Two rules for planning a two-group study. Make the groups the same size whenever the design allows it. Equal n buys the most precision for a fixed total number of participants, and it also makes the test far less sensitive to unequal spreads, which is the pairing that does real damage. Then aim for a combined n₁ + n₂ of at least 40 if the data are likely to be badly skewed, at least 15 in total if both distributions look roughly symmetric, and anything from about 5 per group upward if they look close to normal with no outliers. These bands are the two-sample version of the robustness rule of thumb, and they are more forgiving than the one-sample bands because averaging over two groups smooths two skews instead of one.
Why it matters: comparing two group means is one of the most common tasks in all of research. When you have more than two groups, the move is not a pile of t-tests but a step up to ANOVA, which is where Stats 2 begins. And when it's time to run this test on real thesis data, the guide Analyze Your Thesis Data in JASP walks this exact analysis from CSV import to APA sentence.
The same test on real data: sleep-experiment.csv gives 80 volunteers split across a normal and a restricted night, with t, p and d worked out for you to check against.
Two practice problems run on this lesson: Problem 12 runs the test on twenty-four children, then asks you to judge the researcher who read a non-significant result as proof the two methods work equally well, and Problem 20 works the same test twice, once at the conservative degrees of freedom and once at the Welch ones.
Common questions
What is the difference between Student's t-test and Welch's t-test?
Student's version assumes both groups have equal population variances and pools them; Welch's version drops that assumption and adjusts the degrees of freedom instead. Welch's costs almost nothing when variances are equal and protects you when they're not — which is why many statisticians (and R's default t.test) recommend Welch as the routine choice.
My two groups are men and women, not randomly assigned. Does the t-test still work?
The arithmetic is identical, and that is exactly the trap. A t-test compares two means; whether the difference it finds licenses a causal claim is settled by how people ended up in the groups, not by the test. Randomly assigned groups start out differing only by chance, so a gap afterwards points at what you did to them. Groups you found rather than made differ in all the ways that made them separate groups in the first place, and any of those can be doing the work. Run the test, describe the result as a group difference rather than an effect, and see quasi-experiments and causal DAGs for what can still be recovered when randomization was never available.
How many participants do I need for a t-test?
It depends entirely on the effect size you're trying to detect: with α = .05 and 80% power (two-tailed), a large effect (d = 0.8) needs about 26 per group, a medium one (d = 0.5) about 64, and a small one (d = 0.2) nearly 400. Run the numbers in the power playground before collecting data.