# Under the Hood

By [DYLIT Chronicles](https://dylit.info/user/dylitmediabuzz)

[Everything AI - beyond the hype](https://dylit.info/pr/everything-ai-beyond-the-hype/6a9efac02e92664f4d50cf9d) > [Under the Hood](https://dylit.info/ch/under-the-hood/6ab35f8099638e119f592d1f)

What "AI Hallucination" Actually Is The word is unhelpful. It suggests a malfunction, something going wrong in a system that otherwise reports facts. What is actually happening is the system working exactly as designed, in a situation where its design and your expectations diverge. The mechanism, briefly These models generate text by predicting what comes next, drawing on patterns learned from an enormous quantity of writing. They are not looking anything up. There is no database being queried and no retrieval step. So when you ask for the title of a paper on a topic, the model produces text in the shape of a paper title on that topic. Sometimes the pattern-matching lands on a real paper, because real papers were in the training material and their titles left traces. Sometimes it produces something that has every property of a real title except existing. From inside the mechanism these two cases are identical. There is no internal signal distinguishing "I am reproducing something I encountered" from "I am constructing something plausible." Both are the same operation. That is why hallucination cannot be patched out. It is not a bug sitting next to the useful behaviour. It is the same behaviour. Why it always sounds certain The second half of the problem, and the more dangerous one. Confidence in writing is a stylistic pattern, and the model reproduces it because most of what it learned from is written confidently. Textbooks, articles, documentation and reference material do not hedge every sentence. There is no mechanism connecting tone to reliability. A fabricated figure and a verified one are delivered in the same voice, with the same steadiness. The model does not have a reliable internal sense of uncertainty that surfaces to you as hesitation. This matters because humans read confidence as a signal. Someone speaking clearly and without hedging on a topic usually does know it. That heuristic is well-founded for people and completely broken here. What it looks like in practice The recognisable patterns: Citations that do not exist. Real journal, plausible author, sensible title, right year. Assembled from real components into something that was never published. Numbers with no source. A market size, an adoption rate, a percentage improvement. Presented flatly, often to one decimal place, which makes it read as measured. Policies and features that sound right . A refund window, a statutory notice period, a configuration option in a product. Plausible for that kind of thing, not true of this one. Confident answers about your business. The model knows nothing about your company. Ask it something internal and it will answer from what is typical, presented as though specific. Quotes nobody said. Attributed to a real person, in a register that person plausibly uses. What reduces it You cannot eliminate this. You can substantially narrow it. Give it the source. The single biggest lever. Paste the actual document rather than relying on what the model remembers. Working from text in front of it is a fundamentally different risk profile from working from training. Constrain it explicitly. "Use only the text provided. If something is not covered, say 'not stated' rather than filling it in." This works better than people expect. Permit uncertainty. "If you are not sure, say so." A simple instruction that genuinely changes output, because it makes hedging an acceptable pattern to follow. Require evidence. "Quote the specific line that supports each claim." Now you can check the claim against the quote, and a fabricated claim usually comes with a quote that is not in the document. Frame it as a draft. Asking for something to review produces different output than asking for a fact. The mental adjustment Stop thinking of it as a system that knows things and occasionally errs. Think of it as a system that produces plausible text, which is frequently also true. That reframing changes behaviour in the right direction. You stop expecting reliability and start building verification into how you work, which is the only approach that survives contact with a technology that is confidently wrong at unpredictable intervals. The strange part is that the better these systems get, the more this matters. A model that is wrong ten percent of the time keeps you alert. A model that is wrong one percent of the time, in exactly the same confident voice, is far more likely to get something past you.