Tidy Data
Before any statistics happen, your data has a shape, and the wrong shape is where most analysis pain actually comes from. Tidy data is one simple, boring convention that makes every tool downstream just work: rows are observations, columns are variables, one value per cell. Learn to see the shape, and half of "cleaning" disappears before it begins.
The one rule (that is really three)
Coined by Hadley Wickham, a tidy dataset satisfies three conditions at once:
- Each variable is a column. Height, group, score — one variable, one column, nothing else in it.
- Each observation is a row. One row per unit of analysis (per person, per trial, per person-per-timepoint), consistently.
- Each cell holds one value. No "88/100", no "Ann (Year 2)", no notes crammed in beside the number.
That's it. Tidy data is long and a little dull to look at, but because every tidy table has the same layout, you can filter, group, plot, and model it without rearranging it first. Messy data is messy in a thousand different ways, but tidy data is all tidy in the same way.
The classic spreadsheet sins
Human-friendly spreadsheets are usually machine-hostile. The same handful of sins show up everywhere:
- Metadata living in the data. A title banner, a blank spacer row, a "totals" row mixed in with real observations.
- Two variables in one column.
Ann (Year 2)packs identity and year of study into a single cell. - Color as data. A red-shaded cell that secretly means "resit". No program can compute on a fill color, and neither can future-you.
- One variable spread across many columns.
Quiz 1,Quiz 2,Finalare three columns of the same variable (a score). This "wide" layout is friendly to read and painful to analyze.
Below is a gloriously messy grade sheet with all four. Fix them one at a time and watch it morph into tidy form.
🧹 Fix This Spreadsheet
A messy class grade sheet with four tidy-data sins. Click each sin, newest one first, to repair it. The sheet reshapes one step at a time until every column is a variable, every row an observation, and every cell a single value.
Sins to fix, click the highlighted one:
The end state is taller and less pretty, but every column now answers "what variable is this?" with a single word. You could compute the mean score, the resit rate, or a per-year average with one line of code.
Why tidy unlocks everything downstream
Most analysis tools expect tidy data. A tidy frame drops straight into a descriptives routine, a group_by, a boxplot, or a regression with no reshaping. Once your data is tidy, you stop asking how to get the software to accept it, and the remaining questions are statistical. Reshaping between tidy-long and human-wide is itself a routine operation (pivot_longer and pivot_wider get a lesson of their own later in this course), and it is easy because tidy is a well-defined target.
The tidy test: point at any column and ask "is this one variable?" and point at any row and ask "is this one observation?". If a column name is actually a value (like 2019, Male, or Quiz 1), your data is wide, not tidy: that variable is hiding in your headers.
Why it matters: with tidy data you can rerun an analysis in a second, while a messy sheet has to be rearranged by hand every time. Keep the raw file exactly as collected, and reshape to tidy in a script. The messy original then stays intact, and the tidy version can be rebuilt from scratch by anyone who has the raw file. A good codebook and disciplined data entry are what keep a sheet tidy in the first place.
Common questions
What does tidy data mean?
Tidy data is a simple, standard shape for a table: each variable is a column, each observation is a row, and each cell holds a single value. The term comes from Hadley Wickham. A tidy table is usually long and a little dull to read, but because the layout is always the same, you can filter, group, plot, and model it without reshaping first. Messy data is messy in endless ways, whereas tidy data is all tidy in the same way.
Should data cleaning ever change my raw data file?
Treat the raw file as read-only. Save exactly what you collected, never overwrite it, and do every fix in a script that reads the raw file and writes a separate clean one. That way the messy original stays intact as the ground truth, every change is documented and reversible, and anyone (including future-you) can rerun the whole pipeline from scratch. Hand-editing cells in the raw sheet destroys the record of what actually happened.
My table has one row per person and several measurement columns. Is that untidy?
It depends on what those columns hold. Height, age and reaction time are three different variables, so a person-per-row table carrying them is perfectly tidy. Quiz 1, Quiz 2 and Final are three copies of one variable, score, with the occasion stored in the header, and that table is wide rather than tidy. The test is to read your column names aloud and ask whether each one names a variable or names a value of some other variable. If it's the second, the repair is a single pivot to long and the occasion becomes a column like everything else.