Power Analysis for Complex Designs
You met power in the previous lesson as "the chance of detecting a real effect." This lesson uses it as a planning tool. Before any data is collected, a power analysis tells you how many participants you need for a high chance (usually 80%) of detecting your effect.
The four interlocking quantities
Power analysis links four numbers, and fixing any three determines the fourth. They are the effect size you care about, the sample size (n) you collect, your false-positive rate α (usually 0.05), and power itself, your chance of detecting the effect, which you normally want at 0.80 or above.
The most common use runs it backwards: you choose the effect size you'd hate to miss, set power to 0.80 and α to 0.05, and solve for the sample size. The power curve below does this for two groups, and the power & sample-size calculator uses the same exact calculation for t-tests, ANOVA, correlation, chi-square, and proportions.
🎮 Sample-Size Planner
For a two-group comparison: pick the effect size and the power you want. The curve is power vs. sample size; the marker shows the smallest n per group that reaches your target.
A large effect (d = 0.8) needs only a couple dozen per group, but a small one (d = 0.2) needs hundreds. The required n grows roughly with 1/d², so halving the effect size roughly quadruples the sample. Subtle effects are expensive to detect, and an underpowered study of a small effect tells you very little.
Why "complex designs" are harder
For a simple two-group t-test you can almost read the answer off a table. Most real studies are more complicated, and each complication changes the power calculation:
- More groups / factors (factorial ANOVA). Power depends on which effect you most want to detect, a main effect or an interaction. Interactions usually need much larger samples.
- Repeated measures. The correlation between measurements changes the math, usually boosting power (the within-subjects advantage).
- Multiple comparisons. Correcting α (Bonferroni and similar corrections) lowers the power of each test, so you need more data to compensate.
- Clustered / multilevel data. Students within classrooms aren't independent, which can drastically reduce effective sample size. Thirty classes of 20 give you far less information than 600 independent pupils. How much less depends on the intraclass correlation from multilevel models.
When formulas run out, simulate. For many complex designs there is no closed-form formula. The modern approach is simulation: generate thousands of fake datasets with your assumed effect, run your planned analysis on each, and count how often it comes out significant. That proportion is your power, and the method works for any design you can simulate.
Why it matters: a study that is too small will probably miss the effect it is looking for. One that is needlessly large wastes resources and participants' time. A power analysis sets the sample size between the two, and it combines effect size, sampling, hypothesis testing and design in one practical decision.
Problem 68 of the practice problems takes a methods section that justifies 20 per group as "standard in this literature" and asks what each of the four inputs should actually have been. To answer the same question for your own study, Plan My Analysis steps through your design and gives the sample size along with the test.
Common questions
How do I choose the effect size for a power analysis?
Best: the smallest effect that would still matter (the "smallest effect size of interest"). If you power for it, any effect at least that large has at least your chosen power (say 80%) of being detected. Also common: effects from prior literature or meta-analyses, discounted for publication bias (published effects tend to be inflated, so halving them is a reasonable precaution). Worst: Cohen's "medium" chosen because it's the middle button.
What is post-hoc power analysis, and should I run one?
If it means computing "observed power" from the effect size and n you just obtained, don't. Observed power is a deterministic function of your p-value (p just under .05 always gives power just over 50%), so it adds no information. Journals still occasionally ask for it, but methodologists have long rejected it. A useful after-the-fact question is "what effect size could this design detect with 80% power?"
How big should a pilot study be?
Big enough to test whether the study can run, because a pilot can answer only that question. It is tempting to estimate the effect size from the pilot and power the main study with it, but a pilot cannot support that. An effect estimated from twenty people has an interval so wide that it is compatible with almost any sample size. And if you continue only when the pilot looks promising, its estimate is biased upward as well. A common rule of thumb is around a dozen participants per group. That is enough to find out whether recruitment works, whether the instructions are understood, how long a session takes, and whether your measure has any range at all. Take the effect size for the main study from published work, from theory, or from the smallest effect you would care about (see effect size & power), not from your own pilot.