LLMs & AI in Your Research Workflow
You may have found this site by asking one. Large language models (ChatGPT, Claude, Gemini) can explain a concept, debug your R, and draft a paragraph in seconds, and they'll be an ordinary part of your research life. But the very mechanism that makes them so fluent also makes them structurally prone to making things up. Understand the mechanism and you'll know exactly when to lean on them and when never to.
What an LLM actually does
Strip away the branding and an LLM does one small thing at colossal scale: it predicts the next word. Given the text so far, it produces a probability distribution over what token comes next, samples one, appends it, and repeats. Trained on a vast slice of the internet, it becomes extraordinarily good at continuing text in a way that sounds right. That's all it does, and notice what's not in it: no database of facts, no built-in lookup, no step where it checks whether what it's saying is true. It optimizes for plausibility, not accuracy. The toy below runs the same idea on a tiny, visible corpus so you can watch it happen.
🔮 The prediction machine
This toy has read only the ten sentences below. Press predict next word to watch it pick — each bar is how often a word followed the current one in its tiny "training data". It strings together grammatical text that is often completely false. It isn't lying; it has no idea what true means.
Next-word candidates for "The" — orange = the one it just picked:
🕵️ Spot the fabricated citation
An AI produced these three references for a paper on sleep and memory. One is completely invented. Read them and pick the fake — then see how you did.
Why fluency isn't truth
Because the model is built to produce plausible continuations, a wrong answer that reads smoothly is exactly as easy for it to generate as a right one — often easier. This is hallucination, and it isn't a bug to be patched away; it's a direct consequence of how the thing works. The starkest example is the fabricated citation: asked for references, an LLM will happily invent authors, titles, journals, years, and a plausible-looking DOI, because a citation is just a very patterned string of tokens, and it has learned the pattern without the papers. As the game above shows, the invented reference is frequently the most impressive-looking one — and you cannot tell it's fake by reading it. You have to check.
Two things that complicate this: the cutoff and retrieval
The first is the knowledge cutoff. A model working from its own weights alone answers out of a frozen snapshot of text gathered up to some date. Ask it about a paper published after that date, or about a package whose API changed last month, and it answers anyway, in exactly the same confident register, because nothing in the mechanism separates "I have seen this" from "this is what such an answer usually looks like".
The second is retrieval. Most research-facing AI tools now search the web or a document collection first and paste what they find into the model's context before it writes a word. This genuinely helps: the reply is anchored to text that actually exists, and you get links you can open. What it does not do is transfer the checking to the machine. The model still writes the summary, so it can still misread a retrieved paper, blend two of them, or attach a real citation to a claim that paper never made. A live link beside a sentence is evidence that the source exists. It is never evidence that the sentence is true.
Where AI helps, and where it must not
The mechanism draws a clean line. An LLM is superb wherever you remain the verifier and the output is something you'd check anyway; it is dangerous wherever you'd be tempted to trust it as an oracle.
✅ Strong, legitimate uses
- Explaining a concept or an error message
- Drafting or debugging code you then run
- Critiquing your argument; rubber-duck thinking
- Rewording clunky prose you wrote
- Triaging literature you'll read yourself
⛔ Forbidden / dangerous
- Facts or figures you can't independently verify
- Citations you won't click through and confirm
- Statistics it "computed" that you can't reproduce
- Pasting participant data into an external tool
- Anything you'd present as your own analysis unread
That last red item is also a data-protection issue, not just an accuracy one: participant data pasted into a third-party service has left your control. See Using AI Tools Ethically and Privacy & Confidentiality.
A verification workflow
Treat every AI output as an eager intern's first draft: quick, confident, occasionally brilliant, and never to be trusted unread. Concretely: run any code and check it does what you meant; recompute any number yourself; click through every citation and drop the ones you can't find; and remember the non-negotiable: you own every word and number in your work. "The AI said so" is never a defense, and an AI can't be an author because it can't be accountable (ethics of AI in research). Finally, disclose your use honestly — journal and university policies differ, so check yours.
Why it matters: An LLM is a next-word predictor optimized for plausibility, not truth — so fluency tells you nothing about accuracy, and fabricated facts and citations are structural, not accidental. Used as a tireless assistant whose every output you verify, it's a genuine accelerator for coding, critique, and drafting. Used as an oracle for facts, statistics, or citations you don't check (or as a place to paste participant data), it's a fast route to an error with your name on it. Know the mechanism, keep yourself in the loop, and verify everything.
Common questions
The AI gave me a real citation with a working link. Is the claim safe now?
A working link tells you the source exists, which rules out the crudest failure and not the one that matters. The sentence beside the link is still generated text, so it can misstate the finding, attach a real paper to a claim the paper never made, or quote a number the paper reports for a different subgroup or a different outcome. Checking takes about a minute: open the link, find the sentence that supports the claim, and confirm the figure in the paper matches the figure in the answer. Watch for one specific trap, which is a claim that lives in the source's own citation of an earlier study rather than in its results. That means you have found a reference to a reference and still haven't seen the evidence. The old rule survives retrieval intact: if you cite it, you read it.
Can I use ChatGPT or Claude to run my statistics?
To help you write code that you then run and verify: yes, that is one of the best uses of an AI assistant. To have it report statistical results directly: no. If a chatbot tells you 'the effect was significant, t(48) = 2.31, p = .025,' those numbers may be invented to look plausible, and you have no reproducible trail behind them. Ask instead for R or Python code, run it yourself on your real data, and check the output. You remain responsible for every number in your write-up, so anything you can't reproduce and defend doesn't belong there. Never paste raw participant data into an external tool either — see Using AI Tools Ethically.
Does telling the model "don't make anything up" actually help?
Less than it feels like it should. The instruction is processed the same way as every other word you send: as text conditioning what comes next, not as a switch that turns on a fact-checking step the model does not have. The same goes for asking how confident it is, because that confidence statement is generated by the identical next-word process, which is why a fabricated citation and a real one arrive with equal certainty. What genuinely shifts the odds is giving it something to work from and something you can check: paste the paper or the documentation and ask it to answer only from that text, ask for reasoning in steps you can follow, and ask it to quote the passage it is relying on so you can go looking for that passage. Those help because they make the answer checkable, not because they make the model honest. On when you must declare AI use, see Using AI Tools Ethically.