Hypothesis Testing Logic

A hypothesis test is a structured way to answer "could this result just be a fluke?" The logic mirrors a courtroom: the null hypothesis is presumed innocent, and we only convict (reject it) when the evidence would be surprising if it were true.

The four moves

1. State two hypotheses. The null (H₀) is the boring "nothing's going on" claim (no effect, no difference). The alternative (H₁) is what you suspect instead. (Turning a vague research question into these two crisp statements is a skill of its own. From Question to Hypothesis walks the whole path.)
2. Assume H₀ is true. This gives you a null distribution, the range of results you'd expect from chance alone if there were no effect.
3. Measure surprise. Find where your actual result falls on that distribution. The p-value is the probability of getting a result this extreme or more, just by chance, if H₀ were true.
4. Decide. If the p-value is below a pre-set threshold α (usually 0.05), the result is "too surprising to be chance," so you reject H₀. Otherwise, fail to reject it.

Where the statistic comes from

The explorer below starts from a z, because this lesson is about the logic. An exam gives you a claim and a sample, and the first job is to turn them into that single number. When the population standard deviation σ (sigma) is known, the arithmetic is short. Take the gap between what you saw and what the claim predicts, then measure that gap in standard errors:

z = (x̄ − μ₀) / (σ/√n)

μ₀ (mu-nought) is the value the null hypothesis names, and the question gives it to you. The denominator is the standard error, the usual sample-to-sample variation of a mean. Dividing by it puts z on the same scale whatever the units of measurement: z is the number of standard errors between the sample mean and the claimed value.

What a p-value is — and isn't

Slide the statistic out toward the tails and the p-value shrinks: extreme results are unlikely under chance, so they count as evidence against H₀. But the p-value is easy to misread, and two misreadings are common:

The five steps, in exam order

The four moves above are the logic. On paper, an examiner wants them written in a fixed order with the arithmetic in the middle, and marks are given for the steps rather than the answer:

1. Write H₀ and H₁ as statements about the population parameter, in symbols, before looking at the data. H₀ names one value; H₁ says what "not that value" means for this question.
2. Fix α. It is a decision about how much risk of a false alarm you will accept, and you make it here, before you see the p-value.
3. Compute the test statistic, showing the substitution as well as the result.
4. Find the p-value from a table or a calculator, matching the tail or tails to H₁.
5. Compare p with α and write the conclusion in the words of the original question, not as "reject H₀" alone.

One tail or two, and where the choice comes from

The research question decides the explorer's alternative setting, and it is decided before any data arrive.

Wording that names a direction points at one tail: "is the new fertilizer better", "does the drug lower blood pressure", H₁: μ > μ₀ or H₁: μ < μ₀. Wording that only asks whether something has moved points at both: "is the machine off target", "is the printed label incorrect", "do the two groups differ", H₁: μ ≠ μ₀. Two-sided is the cautious default, and software reports it unless you ask for one tail.

Which one you picked also changes how the p-value is read off a printed table. A left-tailed test reads the area below your z straight from Table A. A right-tailed test reads that area and subtracts it from 1, or equivalently looks up −z and reads the area below that. A two-sided test finds one tail and doubles it.

Significant, and possibly trivial

A small p-value says the data would be surprising if nothing were going on. It says nothing about how much is going on. Sample size separates the two questions: with enough data, a difference far too small to matter will clear any threshold you like.

Picture a weight-loss app tested on forty thousand users, reporting a mean loss 0.2 kg greater than the control group's at p < .001. The difference is almost certainly real, and it is a fifth of a kilogram. Statistical significance is a claim about evidence; practical significance is a claim about size, and only the second one answers "should anyone change what they do?"

To keep them apart, report an effect size and a confidence interval beside every p-value. Then the reader of your paper sees a size and its uncertainty as well as the verdict. If a paper reports an effect size you are not used to, the effect-size converter will convert it to one you know.

Absence of evidence is not evidence of absence

"Study finds no link between X and Y" is a headline written off a p-value above .05, and it almost always claims more than the study showed. Failing to reject H₀ means the data were compatible with no effect. They were also compatible with a small effect, and frequently with a large one, because a test that fails to reject has not measured anything precisely.

The confidence interval tells these cases apart. For a difference, an interval of [−0.1, +0.2] does rule out anything big. An interval of [−4.0, +4.5] rules out nothing at all and reports only that the study was too small to tell. Both results are non-significant, but only the first supports the headline.

Writing About Non-Significant Results covers how to write this up without overclaiming in either direction. It also covers equivalence testing, the procedure to use when "no meaningful difference" is the claim you set out to make.

Two ways to be wrong, and how often

A test can be wrong in two ways. A Type I error rejects a true H₀, reporting an effect that was never there, and its rate is exactly α. So α is the one error rate you choose directly: set it at .05 and you will raise a false alarm on 5% of the occasions when there is no effect. A Type II error misses an effect that is present, and its rate is written β (beta). You do not set β directly. It depends on three things: how big the real effect is, how variable the measurement is, and how much data you gathered.

Studies usually report β's complement. The power of a test, 1 − β, is the probability of detecting an effect of a stated size when that effect exists. Both quantities are easier to see when you draw two sampling distributions on one axis: the one H₀ predicts, and the one the true mean would produce.

A nursery's seedlings have averaged 22.0 cm at four weeks for years, with σ = 0.5 cm. A new light schedule might raise the mean to 22.6. Testing five trays at α = .05, two-sided, the cutoffs are 22.0 ± 1.960 × 0.5/√5, which is 22.0 ± 0.438, so the test rejects for a tray mean below 21.562 or above 22.438. If the new mean is 22.6, the probability that a tray mean is beyond that upper cutoff is .765. With a power of .765, a study designed this way misses a real effect roughly one time in four.

With the two curves in view, each control does what you would expect. More data narrows both curves and reduces their overlap. A larger true effect moves them further apart. A smaller α pushes the cutoffs outward and lowers power. The two error rates trade against each other, and no α setting shrinks both at once. A power analysis chooses n, before any data exist, so that power reaches .80. The power calculator runs one, and Effect Size & Power teaches it in full.

Two practice problems run on this lesson. Problem 9 gives you a newsletter sentence that misreads p = .030 as "a 3% chance the feed makes no difference". Problem 18 runs a σ-known test from start to finish: the tail comes from the wording, and the decision changes between α = .05 and α = .01.