Questionable Research Practices
Most bad science isn't fraud. More often it is a series of small decisions, each defensible on its own, that together turn noise into a "finding": run a few more participants, drop the analysis that didn't work, report the outcome that did. These are questionable research practices (QRPs), the gray zone between honest flexibility and outright fabrication. They are common and feel innocent at the time, but together they mean a p-value no longer has the false-positive rate it claims. This lesson names them, explains why they are tempting, and lets you watch one of them produce a false positive.
The continuum from flexibility to fraud
Picture a line with honest, preregistered analysis at one end and made-up data at the other. QRPs fill the large middle. There, researcher degrees of freedom (all the small choices a dataset lets you make) are used, consciously or not, to reach significance:
- Optional stopping. Peeking at the results as data come in and stopping the moment p < .05 (or running "just a few more" when it isn't). The simulation below shows this one.
- Selective outcome reporting. Measuring five outcomes, then reporting only the one that reached significance, and silently dropping the other four from the write-up.
- HARKing — Hypothesizing After the Results are Known. Finding an unexpected pattern by exploration, then presenting it as though you'd predicted it all along. A chance blip gets relabeled a confirmed theory.
- p-hacking. Trying analyses until one crosses the line (dropping "outliers," adding covariates, testing subgroups, transforming variables) and reporting only the version that worked. This is the garden of forking paths: with enough untracked choices, a false positive is almost guaranteed.
Why they feel innocent
Each step of a QRP comes with a plausible reason. "The result was trending, so I collected a bit more data" sounds like diligence. "That participant clearly wasn't paying attention, so I excluded them" sounds like rigor. "This subgroup effect is fascinating" sounds like insight. None of these feels like cheating, and researchers who do them are usually not being cynical. They are following incentives (publish, get significance) with a dataset flexible enough to reward motivated reasoning. So trying harder to be objective does not fix the problem. Deciding the rules before you see the data does.
How common are they?
They are very common. In a survey of over 2,000 psychologists (John, Loewenstein & Prelec, 2012), around 63% admitted failing to report all of a study's dependent measures. About 56% had decided whether to collect more data after checking if results were significant, roughly 46% had selectively reported studies that "worked," and about 27% admitted HARKing. Outright data falsification was admitted by under 1%. Fraud is rare. The gray-zone practices that inflate false positives are widespread, and they are the main cause of the replication crisis.
Watch optional stopping in action
The simulation below is modeled on the one that made this danger famous (Simmons, Nelson & Simonsohn, 2011). Every "study" in it has no real effect at all: the true difference is exactly zero. An honest researcher fixes the sample size in advance and tests once, and gets a false positive about 5% of the time, as a p < .05 threshold intends. A peeker tests repeatedly as data arrive and stops the instant p dips below .05. Run a thousand of each and compare the false-positive rates.
🎣 The optional-stopping trap
Every study here has a true effect of exactly zero. Peeking still finds "significance."
One example study, running p-value as participants accumulate (true effect = 0):
False-positive rate over 1,000 studies (there is no true effect, so every "hit" is false):
The problem is the extra looks. The peekers in the simulation are not dishonest. They simply gave themselves extra chances to cross the .05 line. Each extra look is another chance of a false positive, so a "5% test" applied five or ten times is no longer a 5% test. The safeguard is to commit to a stopping rule, or to correct for the looks, before the data arrive.
Peeking, done properly
This does not mean you can never look at your data. Clinical trials look often, and they should: a trial that is already harming people has to stop, and a treatment that is already working should not be withheld from the control arm for another year. Those looks are legitimate because the threshold is adjusted for them. In a group-sequential design you fix the number of looks in advance and divide your 5% among them, an approach called alpha spending. The procedure as a whole then makes a false claim 5% of the time, however many looks you planned.
Two spending rules are widely used. Pocock boundaries use a single stricter threshold at every look. O'Brien–Fleming boundaries start out extremely strict and relax toward the end, so stopping early requires overwhelming evidence while the final test stays close to an ordinary one. With five looks in the simulation above (at n = 20, 40, 60, 80, 100), the three rules behave like this:
| Stopping rule | Threshold at each look | False positives |
|---|---|---|
| Peek at .05, stop when it crosses | .05 · .05 · .05 · .05 · .05 | 14.5% |
| Pocock | .015 at every look | 5.0% |
| O'Brien–Fleming | .000005 · .0012 · .0083 · .022 · .041 | 5.0% |
The constant .015 and the rising O'Brien–Fleming sequence are the standard boundaries tabulated in the group-sequential literature.
A sequential design also saves participants. Give the same five looks a real effect to find (d = 0.4, which a single test at the full n = 100 would detect 98% of the time). O'Brien–Fleming still detects it 97.5% of the time and stops at an average of 65 participants. Pocock detects it 95% of the time and stops at an average of 52. So planned looks give nearly the same power with a third to a half fewer participants. Unplanned looks nearly triple the false-positive rate (from 5% to 14.5%), and afterwards you cannot tell whether a given significant result is one of the false ones.
Bayes factors are the other route. The JASP steps for Bayesian estimation tell you to tick Sequential analysis and watch the evidence accumulate as data arrive, and that does not contradict this lesson. A Bayes factor is a running ratio of evidence, not a decision at a fixed line, so monitoring it does not inflate a false-positive rate the way repeated p < .05 tests do. Your stopping rule still belongs in the preregistration, because "collect until BF reaches 6" and "collect 100 people" are different studies, and readers need to know which one you ran.
The fix
QRPs exploit choices made after seeing the data, so the fix is to make those choices before. Preregister your hypotheses, sample size, stopping rule, and primary analysis. Separate confirmatory tests (planned) from exploratory ones, which are fine to run as long as you report them as exploration. If you must monitor data as they come in, use planned interim analyses with corrected thresholds, not ad-hoc peeking. And share materials and data so others can rerun your analysis with the choices you didn't make. None of this bans flexibility. It makes the flexibility visible, so a reader can tell a predicted result from a lucky one.
Why it matters: a single QRP rarely feels like misconduct, and usually isn't. But when a whole field makes the same moves, the literature fills with effects that fail to replicate. Learn the names, then make it a habit to decide your analysis in advance and to report what you actually did, including the analyses that didn't work.
Problem 97 of the practice problems is a manuscript excerpt in which nothing is fabricated and every number is real, and asks you to find the four questionable practices in one short paragraph.
Common questions
Is stopping data collection early always wrong?
No. What matters is when you decided to stop. A planned interim analysis, specified in advance with a corrected significance threshold so the repeated looks are accounted for, is a legitimate and sometimes required design, especially in clinical trials. What inflates false positives is ad-hoc optional stopping: peeking at the results as data arrive and stopping the moment p dips below .05, or running 'just a few more' participants when it hasn't. That turns a nominal 5% test into something closer to 20% or more, because every extra look is another chance to cross the line. Fix your stopping rule before you see the data, and if you must monitor, use a pre-planned procedure with adjusted thresholds, not gut feel.
What is the difference between p-hacking and outright fraud?
They are at different points on a continuum. Fraud (fabricating or falsifying data) means reporting numbers that never came from the study. It is rare (under about 1% admit it) and unambiguous misconduct. p-hacking and other questionable research practices use real data. They exploit researcher degrees of freedom (dropping 'outliers', adding covariates, testing subgroups, optional stopping) until something crosses p < .05, and then report only that. QRPs usually feel like reasonable judgment calls, not cheating. That is why they are so common, and they fill the literature with effects that fail to replicate. Fraud is the rarer offense. QRPs are far more widespread, and they are the main cause of the replication crisis.
I ran the analysis six ways. Which ones do I report?
All six, and the order you ran them in. Running an analysis several ways is a robustness check if the reader sees every version, and p-hacking if they see only the one that worked. The only difference is what you report. Name one analysis as primary before you look at outcomes. Then report the others in a short paragraph or a supplementary table (with covariates and without, with the outliers and without, on the transformed scale and the raw one). If the conclusion holds in all six, that is a much stronger claim than any single result. If it holds in two of six, that is the finding, and leaving out the other four does not change it. The phrase 'the results were unchanged when' should mean that you checked, not that you hope so.