Reliability & Validity
A good measure has to do two different jobs. Reliability is consistency: does it give the same answer when nothing has changed? Validity is accuracy: does it measure what you actually intend? A bathroom scale that always reads 3 kg heavy is perfectly reliable and completely invalid. You need both.
The dartboard picture
The classic metaphor: your true value is the bullseye, and each measurement is a dart. Reliability is how tightly grouped the darts are; validity is whether that group is on the target. Try both sliders below.
🎯 Reliability & Validity Dartboard
Reliability tightens the grouping; validity pulls the group onto the bullseye. The four corners are the four classic combinations, including the sneaky one: consistently wrong.
Flavors of each
Reliability comes in a few forms, depending on what you check for consistency across:
- Test–retest: same people, same test, two occasions: do the scores agree?
- Inter-rater: two observers scoring the same behavior: do they match?
- Internal consistency: do the items within a scale hang together, as if measuring one thing? This is what Cronbach's α quantifies.
Validity also has types: face (does it look right?), construct (does it measure the intended idea?), internal (can we trust the causal claim?), and external (does it generalize beyond the study?). Confound control protects internal validity.
Cronbach's α, built item by item
Internal consistency rises with two things: how strongly the items correlate with each other (r̄), and how many items you have. The standardized formula makes this exact:
α = (k · r̄) ⁄ (1 + (k − 1) · r̄)
🧪 Cronbach's α Simulator
Add items and raise their average inter-item correlation. Even middling items build a reliable scale once there are enough of them (the Spearman–Brown formula, below).
Adding items is not free. α rises with the number of items, so you can always push it up by adding more of them. But α ≥ .90 often means your items are near-duplicates, asking the same question in slightly different words. High internal consistency is not the same as good measurement: a reliable scale can still measure the wrong construct.
Low α has a cost too. Unreliability shrinks every correlation you compute with that measure by a factor of √α. Two scales at α = .70 can therefore correlate at most .70, however closely the underlying constructs are related.
What α means for a single score
Every reliability coefficient comes from a small model. Classical test theory splits each observed score into two pieces. The true score is the value you would get by averaging endless independent measurements of that same person. Measurement error is whatever is left over. The error is taken to be random, so it correlates with nothing and averages out across people. Reliability is then the share of the observed variance that is true-score variance. An α of .90 says nine-tenths of the spread between your respondents is real difference between them, and a tenth of it is noise.
That fraction converts into a margin on a single score. The standard error of measurement is the SD of the errors surrounding one person's true score:
SEM = SD · √(1 − rxx)
Take a test scaled like an IQ score, SD = 15, reliability .90. Then SEM = 15 × √.10 = 4.74 points, and someone who scored 112 has a 95% interval running from 102.7 to 121.3. The score of 112 looks precise, but the band of about nine points either side is closer to what you actually know. Comparing two people is harder still. Both scores include error, so their difference has to reach about 13 points (1.96 × SEM × √2) before you can call it real. Push reliability to .95 and the SEM falls to 3.35, the band to ±6.6. High-stakes tests chase the last few hundredths of α for this reason. A scale at α = .70 (SEM = 8.2, band ±16) is too imprecise to sort individuals at all.
The α simulator above uses the Spearman–Brown prophecy formula. It predicts what a scale's reliability becomes when you lengthen it by a factor of k with items of the same quality. A 10-item scale at α = .70 needs k = 1.71 to reach .80, so 18 items. Reaching .90 takes k = 3.86, or 39 items. Each additional tenth needs more items than the one before, and 39 items is a questionnaire people abandon halfway. The formula also assumes the error is equally large everywhere on the scale. Real tests measure the middle of their range more precisely than the extremes.
Agreement between raters
α answers the internal-consistency question and nothing else. When two people code the same behavior, the statistic you need is Cohen's κ (kappa), and raw percentage agreement is not it. Two coders who both label most clips "on task" will agree much of the time by luck alone, so the question is how far their agreement exceeds chance:
κ = (po − pe) ⁄ (1 − pe)
Here po is the proportion the raters actually agreed on and pe is the proportion they would hit if each kept their own labeling habits but applied them at random. Two coders rating 100 clips who agree on 45 "on task" and 45 "off task" have po = .90 and pe = .50, giving κ = .80. Two coders who agree on 85 "on task" and 5 "off task" also have po = .90, but their shared preference for "on task" lifts pe to .82 and drops κ to .44.
Rare categories lower κ. When one label is uncommon, chance agreement is already high, so κ stays modest even when the coders are doing a decent job. Report percentage agreement and the base rates next to κ so a reader can tell real disagreement from an arithmetic artifact. For ratings on a continuous scale the counterpart is the intraclass correlation. It is the share of the variance due to differences between the targets rather than to rater noise, the same quantity multilevel models use for clustering.
Why it matters: reliability caps validity (a measure can't be accurate if it isn't even consistent) but it never guarantees it. Report both. The most dangerous instrument is the reliable but invalid one, because its repeatable numbers look trustworthy while they consistently miss the true value.
To compute an α yourself, wellbeing-survey.csv has 120 responses to an 8-item Likert scale, two reverse-worded items to catch you out, and the answer to check against.
Whether you reached the right people at all is a separate question from whether your instrument works. Stats 1 covers it in Producing Data & Sampling Design, with a worked example of undercoverage and non-response.
Common questions
What is a good Cronbach's alpha?
The folk rule is α ≥ .70 for research use and ≥ .80 for higher-stakes decisions, but it's a convention, not a law. Counter-intuitively, a very high α (≥ .90) can signal redundant, near-duplicate items rather than a better scale. α also grows with the number of items and depends on your sample, so report it for your own data, and remember it measures internal consistency, not whether you measured the right thing.
My scale's α is .85 but its test–retest correlation is .60. Which one is the reliability?
Both are. A scale has as many reliabilities as there are ways of repeating it, and each one answers a different question. α asks whether the items agree with one another at a single sitting. Test–retest asks whether the same people come out in the same order weeks later. A gap between the two is not a contradiction. When the construct itself changes over time (mood, motivation, state anxiety), a modest retest correlation describes the thing you are measuring, not a fault in the instrument. Report the time between sittings too: a week is short enough for people to recall their own answers, and a year is long enough for them to have changed. Report the coefficient that matches your claim, and use the same one when you compute a standard error of measurement. A stability claim needs the retest value, and an internal-consistency claim needs α.
What counts as a good Cohen's kappa (κ)?
The usual yardstick is Landis and Koch's: .21–.40 fair, .41–.60 moderate, .61–.80 substantial, above .80 almost perfect, with most journals wanting to see .60 or better for coded data. Those bands are a convention someone proposed in 1977, not a result, so read them loosely. How common each category was matters more, because κ is measured against chance agreement. Two coders who agree on 90 of 100 clips get κ = .80 when the two labels are used equally often, but only κ = .44 when one label covers 85 of the clips. Report the raw agreement and the base rates so a reader can tell which situation they are in.