# Under the Hood

By [DYLIT Chronicles](https://dylit.info/user/dylitmediabuzz)

[Everything AI - beyond the hype](https://dylit.info/pr/everything-ai-beyond-the-hype/6a9efac02e92664f4d50cf9d) > [Under the Hood](https://dylit.info/ch/under-the-hood/6ab35f8099638e119f592d1f)

Six Ways to Fact-Check AI Output The worst version of this is not catching an obvious error. It is a colleague catching a plausible one, in a meeting, in front of people, in a document with your name at the top. Checking AI output is not optional diligence you do when there is time. It is a required step in the process, for the same reason proofreading is required: the thing you are producing carries your name whatever produced the first draft. Six practices cover most of it. 1. Treat every specific as unconfirmed Names. Numbers. Dates. Citations. Quotations. Percentages. Statute references. URLs. Job titles. Company names. Every one of those is a specific, and every specific is a claim the model generated rather than retrieved. Some will be right. You cannot tell which by reading. What holds up well, in practice, is the general reasoning and the structure. A model asked to analyse a situation will usually produce a sensible framework with the right considerations in roughly the right order. It is the details hanging off that framework that break, and the details are what get quoted. Useful discipline: read the output once for sense, then read it again with a highlighter, marking every specific. That second read produces your checking list. 2. Check against a primary source Not another AI. Not a summary. The actual thing. If the output cites a study, find the study. If it quotes a policy, open the policy. If it gives a figure from a report, locate the figure in the report. This sounds obvious and is routinely skipped, usually via a shortcut that feels reasonable: searching for the claim and finding something that seems to confirm it. That is weaker than it looks, because a search for a plausible-sounding claim often surfaces adjacent material that does not actually say the thing. 3. Never ask the model to check itself "Are you sure?" and "please verify this" do not do what people think. The model has no separate faculty for checking. Asked to verify, it generates text that fits the pattern of verification, which means it will often produce a confident confirmation of something it invented two messages ago. It may also flip and apologise for a claim that was correct, because apologising is what follows a challenge in most of its training material. The response to "are you sure" tells you nothing about accuracy. It tells you how the model continues text after that question. A related trap: asking for sources after the fact. The model will produce sources, and they may be invented, because a plausible citation is exactly the kind of thing the mechanism is good at producing. 4. Confirm the source exists before confirming what it says Two steps, and people collapse them into one. First: does this source exist at all? Is there a paper with that title by that author in that journal in that year? Fabricated citations are common, and they are convincing because they are assembled from real components. Real journal, plausible author name, sensible title, appropriate year. Second, only once the source is confirmed real: does it actually say what the output claims? A real source misrepresented is more dangerous than an invented one, because it survives a casual check. Someone verifies the paper exists, stops there, and the misattributed claim goes through. 5. Scrutinise hardest where you know least This is the counterintuitive one, and it runs against what everybody actually does. Natural behaviour is to read carefully in your area of expertise, where you are engaged and opinionated, and skim the unfamiliar parts because you have no basis for judgment. That is exactly backwards. In your own field you will catch a mistake without trying. Outside it you have no defences at all, and the output reads just as fluently. Practically: when AI output covers ground you do not know well, that section needs a primary source check or an expert read. It does not get a pass because you cannot evaluate it. That is the reason it needs one. 6. If being wrong would embarrass you, verify it The rule that covers everything the others miss. Not every AI output needs forensic checking. A draft email to a colleague suggesting three meeting times does not require source verification. Proportion matters, and treating everything as high stakes means you will soon treat nothing as high stakes. The test is simple. Imagine this specific claim turning out to be false, in front of the people who will read it. If that thought produces a flinch, check it. That covers anything going to a client, anything a decision rests on, anything in a regulated context, anything that will be quoted, and anything with a number in it that someone might act on. What checking is not It is not reading the output and finding it reasonable. Reasonableness is guaranteed. The system is built to produce text that reads well, which means an error and a fact arrive in identical packaging. A fabricated statistic sits in a sentence looking exactly like a real one, in the same register, with the same confidence. Your instinct that something "seems fine" is calibrated on human writing, where fluency does correlate with care. Here it does not, and the instinct will mislead you every time. Making it stick The habit that works is mechanical rather than attitudinal. Do not try to be more careful in general. Instead, build one concrete step into the process: before anything leaves your hands, highlight every specific and confirm each one against a source. It takes less time than people fear, usually a few minutes, because most documents contain fewer hard specifics than they appear to. And it converts a vague intention to be careful into something you either did or did not do.