I use AI to help me write. I also use it to help me code. I wrote about that last week. This post is the other side. The part I do not say as much about.

The LLM is always confident. Whether the answer is right, half right, or completely made up, it usually comes out the same. Same tone, same formatting, same "here you go" energy. That is the trap. Not that the LLM is bad. The trap is that it does not know when it does not know, and neither do you, until something breaks.

Two kinds of "I do not know"

There is a difference between these two situations:

  1. I know the topic. I use the LLM to draft, to format, to skip the typing. I am the source of truth. The LLM is a faster pen.
  2. I do not know the topic. I use the LLM to learn, to summarise, to answer. The LLM is my only source of truth.

Case 1 is fine. Case 1 is what I wrote about in the "I use AI to write" post. The risk in case 1 is small. If the draft is wrong, I can tell, because I already know the answer.

Case 2 is where people get hurt. If the LLM is the only place I have seen a fact, I have not seen the fact. I have seen a sentence that looks like the fact. That is a different thing.

The problem is, both cases feel the same at the moment you read the response. The LLM does not change its tone. The response does not come with a warning label. You read it, it sounds right, and you move on. The moment where you should be skeptical is invisible. That is the overconfidence.

Hallucination is a percentage, not a switch

People talk about "hallucinations" as if it is a bug that gets fixed. It is not. Every LLM can hallucinate, and that risk is baked into how it works. It generates the most likely next token, not the true next token. Those are different things. Sometimes they agree. Often they do not.

The honest framing: LLM output is not grounded in truth by default, so invented details can slip in very easily. The function does not exist. The API response shape is from the old version. The documentation page was never written. The legal citation is not real. The library name is a plausible mashup of two real ones. This is not a temporary flaw you patch once. It is part of the technology.

So the question is not "does the LLM hallucinate." Of course it does. The question is: what is my process when the LLM is the only source for a fact I am about to act on?

If the answer is "nothing, I just trust it," that is the dangerous position. Not because the LLM is bad. Because the cost of being wrong is not zero, and the LLM is not paying it.

The embarrassing stories

I want to share a few stories from the internet, because they make the point sharper than I can with abstractions.

The lawyer who cited fake cases. In 2023, a lawyer in New York used ChatGPT to draft a legal brief. It looked great. It cited six previous court decisions. None of them existed. The judge was not happy. The lawyer was sanctioned. The cases had realistic names, plausible citations, the right format. A real legal researcher would have spotted it in seconds. The lawyer did not, because the LLM was the only place they had "seen" those cases.
Source: Judge Castel’s sanctions order, Mata v. Avianca (S.D.N.Y. 2023).

The dad who asked ChatGPT about his kid's homework. A widely shared example was a parent asking for help with a maths problem. The LLM gave a wrong answer with full confidence. The funny part is not that the answer was wrong. The funny part is that the LLM explained the wrong answer clearly, step by step, in a way that made it look right. The confidence was the problem.
Source: Andreas Scherbaum’s writeup of ChatGPT inventing a missing angle while helping with a Pythagorean problem.

The package name that did not exist. This one I have hit myself, several times. I ask the LLM to "add a library that does X." It gives me a name. Sometimes it is a real package. Sometimes it is a plausible name for a package that does not exist. I npm install it, it fails, and only then do I find out. The LLM cannot tell me "I made that up." It does not know it made it up. It just produced text that looked right.

The API method I never had to call. A friend was debugging an integration. The LLM told him the response included a metadata.retry_count field. He spent an hour trying to read it from the response. It was not there. The LLM had invented a field that sounded like the kind of field that would exist. He only found out by reading the actual SDK source.
Source: this one I cannot link, it is a friend's debug session, not a public story.

The Stack Overflow answers that were LLM-generated and confidently wrong. Long, well-formatted answers to questions, written in that signature over-helpful LLM tone, can still turn out to be hallucinated. The answer gets upvoted, then the comments fill up with "this is not how this works," then the answer gets deleted. The people who tried the answer in the meantime just lost time.
Source: Stack Overflow’s policy banning generative-AI answers.

Every one of these has the same shape. The output looked authoritative. The person reading it did not have a way to verify it. The mistake was found later, by someone or something else. The cost was paid by the person who trusted it, not the LLM.

What I actually do

I have my own rules for this. The short version: I am the one who runs the migration, who clicks the deploy, who sends the email. The LLM drafts, I verify against the source, I act. The full list of rules is simple enough: write down, before you paste the prompt, where the source of truth has to come from. Then write down, before you act on the response, what the LLM is not allowed to touch. If you cannot answer both, you are about to over-rely.

The trap I keep almost falling into

I want to be honest about one more thing.

The trap is not "the LLM is bad." The trap is that the LLM is good enough that the right process feels like overkill. Most of the time, the answer is right. Most of the time, I do not need to verify. Most of the time, the check is wasted. That is what makes the failure mode dangerous. The cost of overconfidence is rare, but when it lands, it lands hard. The times the LLM is right teach me to skip the check. The few times it is wrong are the ones that matter.

This is the same shape as driving without a seatbelt. You do not need it most of the time. You really need it the one time you do.

The LLM did not change. My overconfidence is on me. The output still arrives with the same tone, the same formatting, the same "here you go." My job is to remember that none of that is evidence of correctness. Evidence of correctness is the source, not the summary.

I am going to keep using the LLM. I am going to keep shipping faster because of it. I am also going to keep being the one who checks, who verifies, who runs the migration, who clicks the deploy. Not because I do not trust the LLM. Because trusting the LLM is not the same as trusting the output. The LLM is consistent. The output is probabilistic. Those are different claims about the world, and I should not conflate them.