Independent Samples t-Test
This is the test you reach for constantly: do two separate groups (treatment vs. control, men vs. women, version A vs. version B) differ on average? Unlike the paired test, the groups are made of different individuals, so we compare their two means head to head.
Signal over noise
The logic is a ratio. On top, the signal: how far apart the two sample means are. On the bottom, the noise: how much the scores bounce around within each group, adjusted for sample size. That ratio is the t-statistic:
t = (x̄₁ − x̄₂) / (standard error of the difference)
A big t, a clear signal sticking out of the noise, means the groups probably really differ. A small t means the gap could easily be chance. Crucially, the same mean difference can be convincing or meaningless depending on the noise and the sample size. The playground makes that tug-of-war visible.
🎮 Two-Group Tug-of-War
Each dot is a person — and the same people stay on screen while you drag the sliders, so you can watch the groups slide apart or spread out. Hit “New samples” for a fresh random draw.
Three levers, one conclusion
- Bigger gap → bigger t. A larger real difference is easier to detect.
- More noise → smaller t. If scores within each group are all over the place, a gap between the means is less convincing.
- Bigger samples → bigger t. More data per group shrinks the standard error, so even a modest gap can become significant.
Try this: set a gap of 8 with high noise and small n — likely not significant. Now slide n up. The means barely move, but the p-value tumbles past 0.05. Same effect, more evidence. This is exactly why sample size and power matter so much.
Report the effect size too. A significant t with a huge sample might reflect a difference too small to care about. Always pair the p-value with Cohen's d — the gap measured in standard deviations — to judge whether the difference actually matters.
What you're assuming
- The two groups are independent (different people, no pairing).
- Each group is roughly normal, or both samples are large enough for the CLT. When neither is true, the Mann-Whitney test asks a closely related question without needing normality at all.
- Similar spread in both groups — and if not, the Welch version of the test relaxes that requirement.
Why it matters: comparing two group means is one of the most common tasks in all of research. When you have more than two groups, the move is not a pile of t-tests but a step up to ANOVA, which is where Stats 2 begins. And when it's time to run this test on real thesis data, the guide Analyze Your Thesis Data in JASP walks this exact analysis from CSV import to APA sentence.
The same test on real data: sleep-experiment.csv gives 80 volunteers split across a normal and a restricted night, with t, p and d worked out for you to check against.
Problem 12 of the practice problems runs this test on twenty-four children, then asks you to judge the researcher who read a non-significant result as proof the two methods work equally well.
Common questions
What is the difference between Student's t-test and Welch's t-test?
Student's version assumes both groups have equal population variances and pools them; Welch's version drops that assumption and adjusts the degrees of freedom instead. Welch's costs almost nothing when variances are equal and protects you when they're not — which is why many statisticians (and R's default t.test) recommend Welch as the routine choice.
My two groups are men and women, not randomly assigned. Does the t-test still work?
The arithmetic is identical, and that is exactly the trap. A t-test compares two means; whether the difference it finds licenses a causal claim is settled by how people ended up in the groups, not by the test. Randomly assigned groups start out differing only by chance, so a gap afterwards points at what you did to them. Groups you found rather than made differ in all the ways that made them separate groups in the first place, and any of those can be doing the work. Run the test, describe the result as a group difference rather than an effect, and see quasi-experiments and causal DAGs for what can still be recovered when randomization was never available.
How many participants do I need for a t-test?
It depends entirely on the effect size you're trying to detect: with α = .05 and 80% power (two-tailed), a large effect (d = 0.8) needs about 26 per group, a medium one (d = 0.5) about 64, and a small one (d = 0.2) nearly 400. Run the numbers in the power playground before collecting data.