Effect Size & Power

A p-value answers one narrow question: is there an effect at all? It says nothing about how big the effect is, or whether your study was even capable of finding it. Those two questions, answered by effect size and power, separate a meaningful result from a misleading one.

Effect size: how big, not just whether

With a huge sample, a trivially small difference can be "statistically significant." With a tiny sample, a huge difference can miss significance. Significance is tangled up with sample size, so we need a measure of the effect that isn't. That's effect size. For a difference between two means, the standard one is Cohen's d: the gap between the means measured in standard deviations.

d = (mean₁ − mean₂) / standard deviation

Rough conventions: d ≈ 0.2 is small, 0.5 medium, 0.8 large. It's the same currency as a z-score (a standardized distance), so it's comparable across studies and scales.

Which standard deviation goes on the bottom? For two groups it is the pooled one, both groups' spreads combined into a single yardstick, which takes for granted that the two spreads are similar to begin with. There is also a small-sample wrinkle worth knowing about: d calculated this way comes out slightly too large when samples are small, by about 4% with ten people per group and 2% with twenty. Multiplying by a correction factor removes that bias and gives you Hedges' g, which is the version meta-analyses pool. Past roughly fifty per group the two agree to two decimal places, so the distinction earns its keep only in small studies.

Power: could you even detect it?

Power is the probability that your study correctly rejects the null when a real effect exists — your chance of not missing it. The convention is to aim for at least 80%. Power analysis is really a four-way negotiation; the playground gives you all four dials:

• Effect size (d). Slide it up and the two populations pull apart and the H₁ curve below slides away from the cutoff: the green area swells. Tiny effects are genuinely hard to detect.
• Sample size (n). The populations don't move an inch, but the sampling curves below sharpen dramatically. Same effect, narrower curves, less overlap across the cutoff → more power. This is the dial you usually control.
• Significance level (α). A stricter α (say .001) drags the cutoff away from zero, shrinking the false-alarm area but eating your power with it. There's no free lunch: α and β trade off.
• One- vs. two-tailed. Committing to a direction moves the cutoff closer to zero and buys power, at the price of being blind to an effect in the other direction.

How precisely do you know d?

The playground treats d as a fact about the world. Your study only ever hands you an estimate of it, and that estimate carries a margin which APA 7 asks you to report. A bare "d = 0.50" reads like a measurement when it is closer to a reading with a range attached.

The range is not d plus or minus something. Because d is a ratio of two quantities you estimated, its sampling distribution is a noncentral t rather than a normal curve, so the interval has to be found by asking which true effects would leave your observed t where it landed. The APA Results Formatter runs that search and writes the answer into the sentence for you; the APA cheat sheet has the finished pattern if you just need the shape of it.

Try it on this playground's own dials. At the defaults, d = 0.50 with 30 per group, the test gives t(58) = 1.94 and p = .058, so the study misses, and the interval on d runs [−0.02, 1.01], which still holds open the possibility that the effect points the other way. Now take the 64 per group the readout above asks for at 80% power, observe the same d = 0.50, and the test gives t(126) = 2.83, p = .005. Its interval is [0.15, 0.85]. The study worked, and it still cannot separate a small effect from a large one.

A result sitting exactly on the significance boundary has a 95% interval whose lower limit is exactly zero, so "significant at .05" and "the interval excludes zero" are two ways of saying one thing. Significance tells you that zero is out, and nothing at all about size. Pinning d down to ±0.20 around 0.50 takes 199 per group, three times what 80% power costs. Detecting an effect and measuring it are different jobs, and the second is the expensive one.

Why underpowered studies are dangerous

If power is only 40%, you'll miss a real effect more often than you find it — and a "non-significant" result tells you almost nothing. Worse, the few significant results that do squeak through tend to overestimate the effect. This is why researchers run a power analysis before collecting data: pick the effect size you care about, the power you want (say 80%), and solve for the sample size you need. Want the exact number for your own study? The power & sample-size calculator does it for t-tests, ANOVA, correlation, and more, and the complete worked project shows the calculation being made for a real design before a single participant is recruited.

Before is doing the work in that sentence. Running the same calculation afterwards on the effect size you happened to observe produces a number that contains no information you didn't already have from the p-value, and power analysis for complex designs sets out why that is arithmetic rather than bad luck, along with the after-the-fact questions that are worth asking.

Problem 9 of the practice problems works through a newsletter's misreading of p = .030, then asks the question this lesson is for: a repeat trial on eight plots comes back p = .21, so has anything been shown?