Writing About Non-Significant Results

Sooner or later a test comes back p = .08, and you will be tempted to call it "a trend toward significance" and move on. Don't. A non-significant result is not a failure, and it is not proof that the effect is zero. It is a specific finding that you can report, and reporting it accurately is a sign of a careful researcher. This lesson explains what these results mean and how to write about them without over- or under-claiming.

"Not significant" is not "no effect"

The most serious mistake is treating p > .05 as evidence that nothing is there. Absence of evidence is not evidence of absence. A high p-value can mean that the effect is near zero, or that your study was too small to detect a real effect (see The Logic of Hypothesis Testing). The p-value alone can't tell those two situations apart. So "we found no effect" is almost always the wrong sentence. What you found is that you couldn't rule the null out, which is a much weaker claim.

There is no "trend toward significance"

"Marginally significant," "approaching significance," "a trend toward…": these phrases present a non-significant result as if it had crossed a line it did not cross. A p of .06 is not "nearly significant": significance is a yes/no threshold you chose in advance, and .06 is on the "no" side. The phrases are also used in only one direction. Researchers do not write that p = .04 "only marginally" reached significance. Report the exact p-value instead: "p = .06" is accurate and informative, while "a trend toward significance" is spin.

What to write instead

A good sentence about a non-significant result includes three things the bare p-value leaves out:

• The effect size and its confidence interval. "d = 0.15, 95% CI [−0.29, 0.59]" tells the reader both your best estimate and how uncertain it is. The CI matters most here, because it shows the whole range of effects your data are compatible with.
• Power or precision context. Was the study big enough to catch the effect you cared about? A wide CI that spans from "meaningfully negative" to "meaningfully positive" is your signal to say the study was underpowered, not that the effect is absent.
• Neutral framing. Write "The difference was not significant," then describe the interval. Don't apologize, add spin, or hide the result.

When you can say "no meaningful effect"

Sometimes a null result is informative: when the confidence interval is narrow and close to zero. If d = 0.05 with a 95% CI of [−0.15, 0.25], you've effectively ruled out anything beyond a small effect, and you're entitled to say the difference is negligible. This is the intuition behind equivalence testing: instead of asking "is the effect exactly zero?", you ask "can I rule out effects big enough to matter?" A precise null result answers yes. A wide one answers "I don't know yet". So the same p > .05 can come from very different situations, and only the CI shows which one you have.

A non-significant result is also not unpublishable, and the study is not wasted. Well-designed null studies correct the record, contribute to meta-analyses, and stop others from chasing effects that aren't there.

Testing for equivalence: TOST

"Rule out effects big enough to matter" is a test you can run. It is called two one-sided tests, or TOST. You supply a bound, and the procedure tells you whether the effect is inside it.

You choose the bound, and you must choose it before you look: the smallest effect size of interest, the point below which you would not care even if the effect were real. It is a substantive decision, not a statistical one, so no formula can give it to you. Suppose you set it at d = 0.3 for the tight null above. TOST then runs two one-sided tests at once, one against "the effect is −0.3 or lower" and one against "the effect is +0.3 or higher", and reports the larger of the two p-values. Reject both and you have evidence that the effect, whatever its sign, is too small to matter.

Run it on the two studies this lesson has been comparing, both of them non-significant:

• The tight one (d = 0.05, 200 per group, 95% CI [−0.15, 0.25]). Against ±0.3 the TOST p is .006. Equivalence declared: the data are incompatible with anything you would call an effect.
• The wide one (d = 0.15, 40 per group, 95% CI [−0.29, 0.59]). Same bound, TOST p = .25. Equivalence cannot be declared. The study cannot rule out a difference that matters, and at this sample size it never could.

A shortcut explains a number that surprises people: at α = .05, TOST succeeds exactly when the 90% confidence interval falls entirely inside your bounds, not the 95% one. Each of the two one-sided tests puts 5% in a single tail, and a 90% interval leaves exactly 5% in each tail. For the tight study that interval is [−0.11, 0.21], well inside ±0.3. For the wide study it runs [−0.22, 0.52] and goes past the upper bound.

You can't run TOST after the fact with a bound chosen to make the numbers work. Set the bound when you set the hypothesis, and say in the Methods that you did. Preregistration & Open Science covers how to record it in advance.

Pick the honest sentence

Here are five studies, all non-significant, that don't all mean the same thing. Read each result card (the p-value, the effect size, and especially the confidence interval), then choose the one sentence that reports it accurately. The others overclaim, dismiss the result, or reach for "a trend."

Problem 100 of the practice problems puts this lesson inside an APA formatting exercise. Three of the four things wrong with its sentence are formatting, and the fourth is the phrase “were more accurate” attached to p = .062.