Questionable Research Practices
Most bad science isn't fraud. It's a series of small, reasonable-feeling decisions, each defensible on its own, that together turn noise into a "finding": run a few more participants, drop the analysis that didn't work, report the outcome that did. These are questionable research practices (QRPs): the gray zone between honest flexibility and outright fabrication. They're common, they feel innocent in the moment, and together they wreck the one number everyone trusts, the p-value. This lesson names them, shows why they're so tempting, and lets you watch one of them manufacture a false positive in real time.
The continuum from flexibility to fraud
Think of a line. At one end, honest, preregistered analysis; at the far end, making up data. QRPs live in the vast middle, where researcher degrees of freedom (all the small choices a dataset lets you make) get used, consciously or not, to chase significance:
- Optional stopping. Peeking at the results as data come in and stopping the moment p < .05 (or running "just a few more" when it isn't). This is the star of the simulation below.
- Selective outcome reporting. Measuring five outcomes, then reporting only the one that reached significance, and silently dropping the other four from the write-up.
- HARKing — Hypothesizing After the Results are Known. Finding an unexpected pattern by exploration, then presenting it as though you'd predicted it all along. A chance blip gets relabeled a confirmed theory.
- p-hacking. Trying analyses until one crosses the line (dropping "outliers," adding covariates, testing subgroups, transforming variables) and reporting only the version that worked. This is the garden of forking paths: with enough untracked choices, a false positive is almost guaranteed.
Why they feel innocent
The dangerous thing about QRPs is that each step has a plausible story. "The result was trending, so I collected a bit more data" sounds like diligence. "That participant clearly wasn't paying attention, so I excluded them" sounds like rigor. "This subgroup effect is fascinating" sounds like insight. None of these feels like cheating, and researchers who do them are usually not being cynical. They're following incentives (publish, get significance) with a dataset flexible enough to reward motivated reasoning. That's exactly why willpower isn't the fix; deciding the rules before you see the data is.
How common are they?
Common enough to be normal. In a survey of over 2,000 psychologists (John, Loewenstein & Prelec, 2012), self-admitted rates were striking: around 63% had failed to report all of a study's dependent measures, about 56% had decided whether to collect more data after checking if results were significant, roughly 46% had selectively reported studies that "worked," and about 27% admitted HARKing. Outright data falsification, by contrast, was admitted by under 1%. That gap is the real lesson: fraud is rare, but the gray-zone practices that inflate false positives are widespread, which is why they, not fraud, are the main engine of the replication crisis.
Watch optional stopping in action
Below is the simulation that made the danger famous (Simmons, Nelson & Simonsohn, 2011). Every "study" in it has no real effect at all: the true difference is exactly zero. An honest researcher fixes the sample size in advance and tests once: they get a false positive about 5% of the time, exactly as a p < .05 threshold promises. A peeker tests repeatedly as data arrive and stops the instant p dips below .05. Run a thousand of each and compare the false-positive rates.
🎣 The optional-stopping trap
Every study here has a true effect of exactly zero. Peeking still finds "significance."
One example study — running p-value as participants accumulate (true effect = 0):
False-positive rate over 1,000 studies (there is no true effect, so every "hit" is false):
The peek is the problem, not the peeker. Nobody in the simulation is dishonest. They just gave themselves extra chances to cross an arbitrary line. Each extra look is another roll of the dice at a false positive, so a "5% test" applied five or ten times stops being a 5% test. The safeguard isn't trying harder to be objective; it's committing to a stopping rule (or correcting for the looks) before the data arrive.
Peeking, done properly
The answer is not "never look." Clinical trials look constantly, and they should: a trial that is already harming people has to stop, and a treatment that is already working should not be withheld from the control arm for another year. What makes those looks legitimate is that the threshold moves with them. In a group-sequential design you fix the number of looks in advance and spend your 5% across all of them, an approach called alpha spending, so the procedure as a whole makes a false claim 5% of the time no matter how often you check.
Two spending rules do most of the work. Pocock boundaries use a single stricter threshold at every look. O'Brien–Fleming boundaries open almost impossibly strict and relax toward the end, so stopping early demands overwhelming evidence while the final test stays close to an ordinary one. Give the simulation above five looks instead of ten (at n = 20, 40, 60, 80, 100) and the three rules behave like this:
| Stopping rule | Threshold at each look | False positives |
|---|---|---|
| Peek at .05, stop when it crosses | .05 · .05 · .05 · .05 · .05 | 14.5% |
| Pocock | .015 at every look | 5.0% |
| O'Brien–Fleming | .000005 · .0012 · .0083 · .022 · .041 | 5.0% |
Those corrected thresholds were not invented for this page. The constant .015 and the rising O'Brien–Fleming sequence reproduce the boundaries tabulated in the group-sequential literature, which is a useful sign that the correction is a standard piece of machinery rather than a hedge.
The reason to plan a sequential design is not penance for wanting to look. Hand the same five looks a real effect to find (d = 0.4, which a single test at the full n = 100 would detect 98% of the time) and O'Brien–Fleming still detects it 97.5% of the time while stopping at an average of 65 participants. Pocock detects it 95% of the time and stops at an average of 52. Planning the looks buys nearly the same answer for roughly half the fieldwork; taking them without planning buys a wrong answer three times too often, with no way afterward to tell which kind you got.
Bayes factors are the other route, and this site's own JASP walkthrough for Bayesian estimation tells you to tick Sequential analysis and watch the evidence accumulate as data arrive. That is not a contradiction of this lesson. A Bayes factor is a running ratio of evidence rather than a verdict at a line, so monitoring it does not inflate a false-positive rate the way repeated p < .05 tests do. Your stopping rule still belongs in the preregistration, because "collect until BF reaches 6" and "collect 100 people" are different studies and a reader is owed the difference.
The fix
Because QRPs exploit choices made after seeing data, the cure is to lock the choices down before: preregister your hypotheses, sample size, stopping rule, and primary analysis; distinguish confirmatory tests (planned) from exploratory ones (fine to run, but reported honestly as exploration); if you must monitor data as it comes in, use planned interim analyses with corrected thresholds rather than ad-hoc peeking; and share materials and data so others can check the forking paths you didn't take. None of this bans flexibility. It just makes it transparent, so a reader can tell a predicted result from a lucky one.
Why it matters: a single QRP rarely feels like misconduct, and usually isn't, but the same move that gets one paper published, repeated across a field, fills the literature with effects that evaporate on replication. Knowing the names is step one; the real habit is deciding your analysis in advance and reporting what you actually did, including the analyses that didn't work.
Problem 40 of the practice problems is a manuscript excerpt in which nothing is fabricated and every number is real, and asks you to name the four questionable practices hiding in one short paragraph.
Common questions
Is stopping data collection early always wrong?
No. What matters is when you decided to stop. A planned interim analysis, specified in advance with a corrected significance threshold so the repeated looks are accounted for, is a legitimate and sometimes required design, especially in clinical trials. What inflates false positives is ad-hoc optional stopping: peeking at the results as data arrive and stopping the moment p dips below .05, or running 'just a few more' participants when it hasn't. That turns a nominal 5% test into something closer to 20% or more, because every extra look is another chance to cross the line. The rule of thumb: fix your stopping rule before you see the data, and if you must monitor, use a pre-planned procedure with adjusted thresholds rather than gut feel.
What is the difference between p-hacking and outright fraud?
They sit at different points on a continuum. Fraud (fabricating or falsifying data) means reporting numbers that never came from the study; it is rare (under about 1% admit it) and unambiguous misconduct. p-hacking and other questionable research practices use real data but exploit researcher degrees of freedom (dropping 'outliers', adding covariates, testing subgroups, optional stopping) until something crosses p < .05, then report only that. QRPs usually feel like reasonable judgment calls rather than cheating, which is exactly why they are so common and so damaging: they fill the literature with effects that evaporate on replication. Fraud is the rarer crime; QRPs are the widespread erosion, and they are the main engine of the replication crisis.
I ran the analysis six ways. Which ones do I report?
All six, and the order you ran them in. Running an analysis several ways is a robustness check when the reader sees every version, and p-hacking when they see the one that worked, so the two are separated entirely by what reaches the page. In practice: name one analysis as primary before you look at outcomes, then report the others as a short paragraph or a supplementary table (with covariates and without, with the outliers and without, on the transformed scale and the raw one). If the conclusion survives all six, saying so is a much stronger claim than any single result. If it survives two of six, that is the finding, and burying it does not make it go away. This is also the honest use of the phrase 'the results were unchanged when', which should mean you checked and not that you hope so.