Designing Surveys & Questionnaires
A survey question is a measuring instrument, and like any instrument it can be miscalibrated. The wording, the answer options, even the order of items can nudge respondents toward an answer they wouldn't otherwise give. The result looks like clean data (numbers, means, percentages), but it's measuring your phrasing, not their views. Good questionnaire design removes those influences so that the answers reflect what respondents actually think.
Wording that tilts the answer
Each item should ask exactly one clear thing, in neutral language, that any respondent can answer honestly. The classic wording problems:
- Leading questions signal the "right" answer: "Don't you agree that…?" nudges people to agree.
- Loaded / emotive wording smuggles in a judgment: asking whether money is being "wasted" presumes it is.
- Double-barreled questions ask two things at once ("the food and the prices"), so anyone who feels differently about the two halves can't answer.
- Double negatives and jargon make people guess what you meant, adding noise: "don't you think it shouldn't…".
Response options matter as much as wording
How you let people answer shapes the answer. For rating scales (Likert items, whose measurement level decides what you may later do with them), the evidence favors 5 to 7 points. Fewer points throw away real distinctions, and more offer precision people can't actually use. The scale should be balanced (as many positive as negative options around a neutral middle), because an all-positive scale (Good / Very good / Excellent) forces even a critic to compliment you. Whether to include a midpoint is a design decision with costs either way. An odd number of points offers a neutral option, which fits people with no opinion but also gives unmotivated respondents an easy answer. An even number forces a lean. And label the points with words, not just numbers: people interpret "4" very differently, but everyone reads "Agree" the same way.
When everyone picks the top box
A response scale can also fail by running out of room. When most respondents choose the highest option, the item has a ceiling effect; when they pile onto the lowest, a floor effect. Either way the item stops separating people who differ, and nothing in the file shows it. The file looks complete and the means look reasonable, but the information was never recorded.
You can measure the cost. Suppose two constructs correlate .50 in the population. Measure each with a balanced 5-point item, one where about 12% of people pick the top box, and the observed correlation comes out at .46. Cutting a continuum into five categories loses .04. Now reword so that two-thirds pick the top box. The same underlying .50 now comes out as .39. Group differences shrink the same way. A true gap of half a standard deviation shows as d = 0.48 on the balanced item and d = 0.38 once the control group is pressed against the ceiling. At 60 people per group that drops power from .74 to .54, so an otherwise sound study misses a real effect close to half the time, and nothing in the output explains why.
Check the frequency table before you compute anything else. Print the distribution of every item and look at the end boxes. A top category holding a quarter of your sample, or a mean within about one SD of the maximum, is the warning sign. Fix it at the design stage, because no transformation restores a distinction the answer options never captured. Add items that discriminate at the crowded end, widen the range, or move the anchors so the middle of your actual population is near the middle of the scale. This is the same damage restriction of range does to a correlation, caused here by the response options rather than by the sample.
Fix this survey
Below is a very bad six-item questionnaire. Each item has one main flaw. Pick the flaw for each item to see the repaired version.
🔧 Diagnose & Repair the Questionnaire
A student union survey full of problems. For each item, click its main flaw. Get it right and the fixed version appears. Get all six and the repaired survey appears below.
Flaws found: 0 / 6
✔ The repaired survey
The subtler biases
Even clean items can be undermined by the survey as a whole:
- Order effects: an earlier question can prime a later one. Ask about a specific worry, then overall happiness, and happiness drops. The worry is now top of mind. Put general questions before specific ones, and randomize order where you can.
- Acquiescence bias: some people just agree with whatever you say. The defense is reverse-coded items (a few worded in the opposite direction), so straight-line "agree" responding contradicts itself and can be caught. (You then flip those items back before scoring; this ties directly to a scale's internal consistency.)
- Social desirability bias: on sensitive topics like cheating, drinking or prejudice, people answer to look good rather than truthfully. Guaranteeing (and clearly stating) anonymity, softening the framing, and never attaching identifiers to sensitive items all reduce it.
Always pilot. Before a questionnaire goes live, have a handful of people from the target group fill it in and talk you through their thinking (a "cognitive interview"). You'll often find that a question you thought was clear is read three different ways. That is cheap to fix now and impossible to fix after the data is collected. Every item is an operationalization of a construct, and piloting checks that it measures what you meant. Design with the cleaning in mind, too. The reverse-coded items, the "other" boxes and the optional questions you add here are what someone has to untangle later, and the clean-survey-data guide works through that on a real export.
Why it matters: a biased question produces biased data, and no statistic can remove that bias afterward.
Problem 96 of the practice problems is a council survey that collected 4,300 responses and still could not answer its own question, because its problems were all in the design.
Common questions
Should I make every question required?
Forcing an answer does not create information. It moves the problem somewhere else. A respondent who cannot answer honestly will either pick something arbitrary or abandon the survey. An arbitrary answer is noise you cannot distinguish from data. Abandoning turns an item-level gap into a whole missing person and biases the sample toward people who did not mind the question. Both are worse than a blank. Make an item required only where the answer is necessary and everyone you are asking can answer it. Give the others an explicit way out, a 'prefer not to say' or 'not applicable' option rather than a silent skip, so you can tell refusal from oversight. Record which is which in your codebook. You are deciding the missingness mechanism you will have to defend later, and a forced-choice survey with heavy dropout has chosen the worst one without meaning to.
What is a double-barreled question?
A double-barreled question asks about two things in a single item, so one answer can't cover both. For example, 'How satisfied are you with the pay and the working hours?' Someone happy with the hours but not the pay has no valid response, and you can't tell which half their answer refers to. The fix is simple: split it into two separate questions, one per idea. The same rule applies whenever an item contains an 'and' or an 'or' whose parts respondents might answer differently.
What is social desirability bias, and how do you reduce it?
Social desirability bias is the tendency to answer sensitive questions in a way that makes one look good rather than truthfully: under-reporting cheating or drinking, over-reporting voting or exercise. It affects the topics on which researchers most need accurate data. You reduce it by guaranteeing and clearly stating anonymity, never attaching identifiers to sensitive items, and softening the framing so the undesirable answer feels acceptable. For the most delicate questions, use indirect techniques such as list experiments.