LLMs & AI in Your Research Workflow
You may have found this site by asking one. Large language models (ChatGPT, Claude, Gemini) can explain a concept, debug your R code and draft a paragraph in seconds, and they will be an ordinary part of your research. But the mechanism that makes them fluent also makes them prone to making things up, and that tendency is built into how they work. Once you understand the mechanism, you will know when to rely on them and when not to.
What an LLM actually does
An LLM does one simple thing on a very large scale: it predicts the next word. Given the text so far, it produces a probability distribution over the next token (a word or part of a word), samples one, appends it, and repeats. Trained on a large share of the text on the internet, it becomes very good at continuing text in a way that sounds right. Nothing else is in the loop. There is no database of facts, no lookup step, and no step that checks whether the text is true. It is trained for plausibility, not accuracy. The toy below runs the same idea on a tiny corpus you can read, so you can see it happen.
🔮 The prediction machine
This toy has read only the ten sentences below. Press predict next word to see it choose one. Each bar shows how often a word followed the current one in its tiny "training data". It strings together grammatical text that is often completely false. It is not lying. It has no representation of truth at all, only of which words follow which.
Next-word candidates for "The". The orange bar is the one it just picked:
🕵️ Spot the fabricated citation
An AI produced these three references for a paper on sleep and memory. One is completely invented. Read them and pick the fake, then see how you did.
Why fluency isn't truth
Because the model is built to produce plausible continuations, a wrong answer that reads smoothly can be as easy for it to generate as a right one, and sometimes easier. This is called hallucination. It follows directly from how the model works, so no patch can remove it completely. The clearest example is the fabricated citation. Asked for references, an LLM will readily invent authors, titles, journals, years and a plausible-looking DOI. A citation is a very regular string of tokens, and the model has learned the pattern without the papers. As the game above shows, the invented reference can be the most impressive-looking one, and you cannot tell it is fake by reading it. You have to check.
Two things that complicate this: the cutoff and retrieval
The first is the knowledge cutoff. A model working from its own weights alone has learned only from text gathered up to some date. Ask it about a paper published after that date, or about a package whose functions changed last month, and it answers anyway, in the same confident tone. Nothing in the mechanism separates "I have seen this" from "this is what such an answer usually looks like".
The second is retrieval. Most AI tools aimed at researchers now search the web or a document collection first, and add what they find to the model's input before it writes anything. This helps: the reply is based on text that actually exists, and you get links you can open. But the model still writes the summary, so it can misread a retrieved paper, mix up two of them, or attach a real citation to a claim that paper never made. A working link beside a sentence is evidence that the source exists. It is not evidence that the sentence is true.
Where AI helps, and where it must not
The mechanism tells you where to draw the line. An LLM is very useful wherever you do the checking and the output is something you would check anyway. It is dangerous wherever you would be tempted to trust it without checking.
✅ Strong, legitimate uses
- Explaining a concept or an error message
- Drafting or debugging code you then run
- Critiquing your argument; rubber-duck thinking
- Rewording clunky prose you wrote
- Triaging literature you'll read yourself
⛔ Forbidden / dangerous
- Facts or figures you can't independently verify
- Citations you won't click through and confirm
- Statistics it "computed" that you can't reproduce
- Pasting participant data into an external tool
- Anything you'd present as your own analysis unread
Pasting participant data is a data-protection problem as well: data pasted into a third-party service have left your control. See Using AI Tools Ethically and Privacy & Confidentiality.
A verification workflow
Treat every AI output as a first draft that you check before you use it. Run any code and check that it does what you meant. Recompute any number yourself. Click through every citation and drop the ones you can't find. And remember that you are responsible for every word and number in your work. "The AI said so" is never a defense, and an AI can't be an author because it can't be held accountable (ethics of AI in research). Finally, disclose how you used AI. Journal and university policies differ, so check yours.
Why it matters: an LLM is a next-word predictor trained for plausibility, not truth. So fluency tells you nothing about accuracy, and invented facts and citations are a built-in risk, not a rare accident. If you check everything it produces, it can save you a lot of time on coding, critique and drafting. If you rely on it for facts, statistics or citations you don't check, or paste participant data into it, you risk putting your name on its errors.
Common questions
The AI gave me a real citation with a working link. Is the claim safe now?
A working link tells you the source exists. That rules out an invented reference, but not a misrepresented one. The sentence beside the link is still generated text, so it can misstate the finding, attach a real paper to a claim the paper never made, or quote a number the paper reports for a different subgroup or outcome. Checking takes about a minute: open the link, find the sentence that supports the claim, and confirm that the figure in the paper matches the figure in the answer. Watch for one trap in particular: a claim that comes from the source's own citation of an earlier study, not from its results. Then you have found a reference to a reference, and you still haven't seen the evidence. The old rule still applies when a tool retrieves sources for you: if you cite it, you read it.
Can I use ChatGPT or Claude to run my statistics?
Use it to help you write code that you then run and check, which is one of the best uses of an AI assistant. Do not use it to report statistical results directly. If a chatbot tells you 'the effect was significant, t(48) = 2.31, p = .025,' those numbers may be invented to look plausible, and there is no analysis behind them that you can rerun. Ask for R or Python code, run it yourself on your real data, and check the output. You are responsible for every number in your write-up, so leave out anything you cannot reproduce and defend. And never paste raw participant data into an external tool (see Using AI Tools Ethically).
Does telling the model "don't make anything up" actually help?
Less than you might expect. The instruction is processed like every other word you send: it is text that shapes what comes next. It does not switch on a fact-checking step, because the model has none. The same applies when you ask how confident it is. The confidence statement comes from the same next-word process, so a fabricated citation and a real one are stated with equal certainty. What does improve the odds is giving it something to work from and something you can check. Paste the paper or the documentation and ask it to answer only from that text, ask for reasoning in steps you can follow, and ask it to quote the passage it relies on so you can find that passage yourself. These help because they make the answer checkable. They do not make the model more truthful. On when you must declare AI use, see Using AI Tools Ethically.