Effect Size & Power

A p-value answers one narrow question: is there an effect at all? It says nothing about how big the effect is, or whether your study was even capable of finding it. Effect size answers the first of those questions and power the second.

Effect size: how big, not just whether

With a huge sample, a trivially small difference can be "statistically significant." With a tiny sample, a huge difference can miss significance. Because significance depends on sample size, you also need a measure of the effect that does not: an effect size. For a difference between two means, the standard one is Cohen's d: the gap between the means measured in standard deviations.

d = (mean₁ − mean₂) / standard deviation

Rough conventions: d ≈ 0.2 is small, 0.5 medium, 0.8 large. Like a z-score, d is a standardized distance, so it can be compared across studies and scales.

Which standard deviation goes on the bottom? For two groups it is the pooled one, which combines both groups' spreads into a single SD. Pooling assumes the two spreads are similar to begin with. In small samples, d calculated this way also comes out slightly too large: by about 4% with ten people per group and 2% with twenty. Multiplying by a correction factor removes that bias and gives you Hedges' g, the version meta-analyses pool. Past roughly fifty per group the two agree to two decimal places, so the correction matters only in small studies.

Power: could you even detect it?

Power is the probability that your study correctly rejects the null when a real effect exists — your chance of not missing it. The convention is to aim for at least 80%. Four things set a study's power, and the playground has a control for each:

• Effect size (d). Slide it up and the two populations pull apart and the H₁ curve below slides away from the cutoff: the green area swells. Tiny effects are hard to detect.
• Sample size (n). The populations do not move, but the sampling curves below become much narrower. With the same effect, narrower curves overlap less across the cutoff, so power rises. Sample size is usually the one you control.
• Significance level (α). A stricter α (say .001) moves the cutoff away from zero. That shrinks the false-alarm area but also lowers power, so α and β trade off.
• One- vs. two-tailed. Committing to a direction moves the cutoff closer to zero and raises power, but the test can then never detect an effect in the other direction.

How precisely do you know d?

The playground treats d as a fact about the world. A study only gives you an estimate of it, and APA 7 asks you to report a confidence interval around that estimate. A bare "d = 0.50" hides how wide that interval can be.

The interval is not simply d plus or minus a fixed amount. Because d is a ratio of two quantities you estimated, its sampling distribution is a noncentral t rather than a normal curve. To find the interval, you search for the true effects under which your observed t would not fall in either 2.5% tail. The APA Results Formatter runs that search and writes the interval into your results sentence. The APA cheat sheet shows the finished sentence pattern.

Try it with the playground's settings. At the defaults, d = 0.50 with 30 per group, the test gives t(58) = 1.94 and p = .058, so the study misses. The interval on d runs [−0.02, 1.01], so an effect in the other direction is still possible. Now take the 64 per group that the readout above gives for 80% power and observe the same d = 0.50. The test gives t(126) = 2.83, p = .005, and the interval is [0.15, 0.85]. The study detected the effect, but it still cannot tell a small effect from a large one.

A result exactly on the significance boundary has a 95% interval whose lower limit is exactly zero. So "significant at .05" and "the interval excludes zero" say the same thing. Significance tells you that zero is outside the interval, and nothing about size. Estimating d to within ±0.20 of 0.50 takes 199 per group, about three times the 64 that 80% power needs.

Why underpowered studies are dangerous

If power is only 40%, you'll miss a real effect more often than you find it, and a "non-significant" result tells you almost nothing. Worse, the few significant results that do squeak through tend to overestimate the effect. This is why researchers run a power analysis before collecting data: pick the effect size you care about, the power you want (say 80%), and solve for the sample size you need. Want the exact number for your own study? The power & sample-size calculator does it for t-tests, ANOVA, correlation, and more. The complete worked project shows the calculation for a full study design, made before any participant is recruited.

The calculation has to come before the data. Run it afterwards on the effect size you happened to observe, and the power it reports tells you nothing the p-value did not. Power analysis for complex designs explains why, and which questions you can usefully ask after the study.

Problem 9 of the practice problems works through a newsletter's misreading of p = .030. It then asks the question this lesson answers: a repeat trial on eight plots gives p = .21, so has anything been shown?