Power Analysis for Complex Designs
You met power back in Stats 1 as "the chance of detecting a real effect." Here we put it to work as a planning tool. The single most valuable thing statistics can do for a study happens before any data is collected: telling you how many participants you'll need to give your effect a fair chance of being found.
The four interlocking quantities
Power analysis links four numbers. Fix any three and the fourth is determined:
- Effect size: how big the effect you care about is.
- Sample size (n): how much data you collect.
- α: your false-positive rate (usually 0.05).
- Power: your chance of detecting the effect (target: ≥ 0.80).
The most common use runs it backwards: you choose the effect size you'd hate to miss, set power to 0.80 and α to 0.05, and solve for the sample size. The power curve below does exactly that, and the power & sample-size calculator runs the same exact math for t-tests, ANOVA, correlation, chi-square, and proportions.
🎮 Sample-Size Planner
For a two-group comparison: pick the effect size and the power you want. The curve is power vs. sample size; the marker shows the smallest n per group that reaches your target.
Watch the punishing math of small effects. A large effect (d = 0.8) needs only a couple dozen per group; a small one (d = 0.2) can demand hundreds. Because power depends on effect size and n together, halving the effect you're chasing roughly quadruples the sample you need. This is why detecting subtle effects is so expensive, and why underpowered studies of small effects are nearly worthless.
Why "complex designs" are harder
For a simple two-group t-test you can almost read the answer off a table. Real studies are messier, and each wrinkle changes the power calculation:
- More groups / factors (factorial ANOVA) — power depends on which specific effect (a main effect? an interaction?) you most want to detect; interactions are notoriously power-hungry.
- Repeated measures — the correlation between measurements changes the math, usually boosting power (the within-subjects advantage).
- Multiple comparisons — correcting α (Bonferroni and friends) lowers per-test power, so you need more data to compensate.
- Clustered / multilevel data — students within classrooms aren't independent, which can drastically reduce effective sample size. Recruiting 30 classes of 20 buys you far less than 600 independent pupils, and how much less depends on the intraclass correlation from multilevel models.
When formulas run out, simulate. For genuinely complex designs there's often no clean formula. The modern approach is simulation: generate thousands of fake datasets with your assumed effect, run your planned analysis on each, and count how often it comes out significant. That proportion is your power — and it works for any design you can simulate.
Why it matters: a study that's too small wastes everyone's time and can't find what it's looking for; one that's needlessly huge wastes resources and over-tests participants. Power analysis is how responsible research is planned, and it's the perfect place to end, because it brings together effect size, sampling, hypothesis testing, and design into one practical decision.
Problem 28 of the practice problems takes a methods section that justifies 20 per group as "standard in this literature" and asks what each of the four inputs should actually have been. If you would rather answer it for your own study, Plan My Analysis walks the design and returns the sample size alongside the test.
Common questions
How do I choose the effect size for a power analysis?
Best: the smallest effect that would still matter (the "smallest effect size of interest") — powering for it means anything you miss was too small to care about. Also common: effects from prior literature or meta-analyses, discounted for publication bias (published effects run inflated; halving them is not paranoid). Worst: Cohen's "medium" chosen because it's the middle button.
What is post-hoc power analysis, and should I run one?
If it means computing "observed power" from your just-obtained effect size and n — don't. Observed power is a deterministic function of your p-value (p just under .05 always gives power just over 50%), so it adds literally no information; journals still occasionally ask, but methodologists have thoroughly debunked it. Meaningful after-the-fact questions sound like: "what effect size could this design detect with 80% power?"
How big should a pilot study be?
Big enough to test whether the study can run, and not one participant more, because that is the only question a pilot can answer. The tempting move is to read an effect size off the pilot and power the real study with it, and that is exactly what a pilot cannot support: an effect estimated from twenty people carries an interval so wide that it is compatible with almost any sample-size answer, and if the pilot looked promising enough to continue then its estimate is selected upward as well. A common rule of thumb is around a dozen participants per group, which is enough to find out whether recruitment works, whether the instructions are understood, how long a session really takes, and whether your measure has any range at all. Get the effect size for the main study from published work, from theory, or from the smallest effect you would care about (see effect size & power) rather than from your own pilot.