Tidy Data
Before any statistics happen, your data has a shape, and the wrong shape is where most analysis pain actually comes from. Tidy data is one simple, boring convention that makes every tool downstream just work: rows are observations, columns are variables, one value per cell. Learn to see the shape, and half of "cleaning" disappears before it begins.
The one rule (that is really three)
Coined by Hadley Wickham, a tidy dataset satisfies three conditions at once:
- Each variable is a column. Height, group, score — one variable, one column, nothing else in it.
- Each observation is a row. One row per unit of analysis (per person, per trial, per person-per-timepoint), consistently.
- Each cell holds one value. No "88/100", no "Ann (Year 2)", no notes crammed in beside the number.
That's it. Tidy data is long and a little dull to look at, and that dullness is exactly what lets you filter, group, plot, and model it without wrestling the layout first. Messy data is messy in a thousand creative ways; tidy data is all tidy in the same way.
The classic spreadsheet sins
Human-friendly spreadsheets are usually machine-hostile. The same handful of sins show up everywhere:
- Metadata living in the data. A title banner, a blank spacer row, a "totals" row mixed in with real observations.
- Two variables in one column.
Ann (Year 2)packs identity and year of study into a single cell. - Color as data. A red-shaded cell that secretly means "resit". No program can compute on a fill color, and neither can future-you.
- One variable spread across many columns.
Quiz 1,Quiz 2,Finalare three columns of the same variable (a score). This "wide" layout is friendly to read and painful to analyze.
Below is a gloriously messy grade sheet with all four. Fix them one at a time and watch it morph into tidy form.
🧹 Fix This Spreadsheet
A messy class grade sheet with four tidy-data sins. Click each sin, newest one first, to repair it — the sheet reshapes one step at a time until every column is a variable, every row an observation, and every cell a single value.
Sins to fix, click the highlighted one:
Notice the end state: it's taller and less pretty, but every column now answers "what variable is this?" with a single word, and you could compute the mean score, the resit rate, or a per-year average with one line of code. That's the whole payoff.
Why tidy unlocks everything downstream
Tidy data is the shared language your tools already speak. A tidy frame drops straight into a descriptives routine, a group_by, a boxplot, or a regression with no reshaping. The moment your data is tidy, "how do I get the software to accept this?" stops being a question, and the questions become statistical again. Reshaping between tidy-long and human-wide is itself a routine operation (pivot_longer and pivot_wider get a lesson of their own later in this course), but it's only painless because tidy is the well-defined destination.
The tidy test: point at any column and ask "is this one variable?" and point at any row and ask "is this one observation?". If a column name is actually a value (like 2019, Male, or Quiz 1), your data is wide, not tidy — that variable is hiding in your headers.
Why it matters: tidy data is not fussiness for its own sake. It's the difference between an analysis you can rerun in a second and one you rebuild by hand every time. Keep the raw file exactly as collected, then reshape to tidy in a script, so the messy original stays intact and the tidy version can be rebuilt from scratch by anyone who has the raw file. A good codebook and disciplined data entry are what keep a sheet tidy in the first place.
Common questions
What does tidy data mean?
Tidy data is a simple, standard shape for a table: each variable is a column, each observation is a row, and each cell holds a single value. The term comes from Hadley Wickham. A tidy table is usually long and a little dull to read, but that consistency is exactly what lets you filter, group, plot, and model it without reshaping first — messy data is messy in endless ways, whereas tidy data is all tidy in the same way.
Should data cleaning ever change my raw data file?
Treat the raw file as read-only. Save exactly what you collected, never overwrite it, and do every fix in a script that reads the raw file and writes a separate clean one. That way the messy original stays intact as the ground truth, every change is documented and reversible, and anyone (including future-you) can rerun the whole pipeline from scratch. Hand-editing cells in the raw sheet destroys the record of what actually happened.
My table has one row per person and several measurement columns. Is that untidy?
It depends on what those columns hold. Height, age and reaction time are three different variables, so a person-per-row table carrying them is perfectly tidy. Quiz 1, Quiz 2 and Final are three copies of one variable, score, with the occasion smuggled into the header, and that table is wide rather than tidy. The test is to read your column names aloud and ask whether each one names a variable or names a value of some other variable. If it's the second, the repair is a single pivot to long and the occasion becomes a column like everything else.