Reliability & Validity
A good measure has to do two different jobs. Reliability is consistency: does it give the same answer when nothing has changed? Validity is accuracy: does it measure what you actually intend? A bathroom scale that always reads 3 kg heavy is perfectly reliable and completely invalid. You need both.
The dartboard picture
The classic metaphor: your true value is the bullseye, and each measurement is a dart. Reliability is how tightly grouped the darts are; validity is whether that group sits on the target. Play with both dials below.
🎯 Reliability & Validity Dartboard
Reliability tightens the grouping; validity pulls the group onto the bullseye. The four corners are the four classic combinations — including the sneaky one: consistently wrong.
Flavors of each
Reliability comes in a few forms, depending on what you check for consistency across:
- Test–retest: same people, same test, two occasions: do the scores agree?
- Inter-rater: two observers scoring the same behavior — do they match?
- Internal consistency: do the items within a scale hang together, as if measuring one thing? This is what Cronbach's α quantifies.
Validity also has types: face (does it look right?), construct (does it really tap the intended idea?), internal (can we trust the causal claim, which is what confound control protects), and external (does it generalize beyond the study?).
Cronbach's α, built item by item
Internal consistency rises with two things: how strongly the items correlate with each other (r̄), and how many items you have. The standardized formula makes this exact:
α = (k · r̄) ⁄ (1 + (k − 1) · r̄)
🧪 Cronbach's α Simulator
Add items and dial up their average inter-item correlation. Even middling items build a reliable scale once there are enough of them. That's the Spearman–Brown insight.
More items is not a free lunch. The formula rewards length, so you can always crank α up by padding the scale — but α ≥ .90 often means your items are near-duplicates, asking the same question in slightly different words. High internal consistency is not the same as good measurement; a reliable scale can still be measuring the wrong construct.
Low α costs you as well, and the bill arrives later. Unreliability shrinks every correlation you compute with that measure by a factor of √α, so two scales at α = .70 can report at most .70 between them however tightly the underlying constructs are related.
Agreement between raters
α answers the internal-consistency question and nothing else. When two people code the same behavior, the statistic you need is Cohen's κ (kappa), and raw percentage agreement is not it. Two coders who both label most clips "on task" will agree much of the time by luck alone, so the honest question is how far above chance they landed:
κ = (po − pe) ⁄ (1 − pe)
Here po is the proportion the raters actually agreed on and pe is the proportion they would hit if each kept their own labeling habits but applied them at random. Two coders rating 100 clips who agree on 45 "on task" and 45 "off task" have po = .90 and pe = .50, giving κ = .80. Two coders who agree on 85 "on task" and 5 "off task" also have po = .90, but their shared preference for "on task" lifts pe to .82 and drops κ to .44. Identical raw agreement, very different evidence.
Rare categories punish κ. When one label is uncommon, chance agreement is already high, so κ stays modest even when the coders are doing a decent job. Report percentage agreement and the base rates next to κ so a reader can tell real disagreement from an arithmetic artifact. For ratings on a continuous scale the counterpart is the intraclass correlation, the same "share of variance that is genuine between-target difference rather than rater noise" quantity that multilevel models use for clustering.
Why it matters: reliability caps validity (a measure can't be accurate if it isn't even consistent) but it never guarantees it. Report both, and remember the dartboard: the most dangerous instrument is the reliable, invalid one, because its tidy, repeatable numbers look trustworthy while pointing steadily at the wrong answer.
To compute an α rather than read about one, wellbeing-survey.csv ships 120 responses to an 8-item Likert scale, two reverse-worded items to catch you out, and the answer to check against.
Common questions
What is a good Cronbach's alpha?
The folk rule is α ≥ .70 for research use and ≥ .80 for higher-stakes decisions, but it's a convention, not a law. Counter-intuitively, a very high α (≥ .90) can signal redundant, near-duplicate items rather than a better scale. α also grows with the number of items and depends on your sample, so report it for your own data, and remember it measures internal consistency, not whether you measured the right thing.
Can a measure be reliable but not valid?
Absolutely, and it's the most dangerous case. A bathroom scale that always reads 3 kg heavy is perfectly reliable (it's consistent) yet completely invalid (it's systematically wrong). Reliability is necessary for validity — a measure that can't even agree with itself can't be accurate — but it never guarantees it. A precise, repeatable number can still be measuring the wrong thing.
What counts as a good Cohen's kappa (κ)?
The usual yardstick is Landis and Koch's: .21–.40 fair, .41–.60 moderate, .61–.80 substantial, above .80 almost perfect, with most journals wanting to see .60 or better for coded data. Those bands are a convention someone proposed in 1977, not a result, so read them loosely. What matters more is how common each category was, because κ is scored against chance agreement: two coders who agree on 90 of 100 clips earn κ = .80 when the two labels are used equally often, but only κ = .44 when one label covers 85 of the clips. Same raters, same 90%, half the credit. Report the raw agreement and the base rates so a reader can tell which situation they are in.