Why AI Chatbots Make Things Up: Guessing Scores Better

Because a guess scores better than an admission. OpenAI’s own account is that models hallucinate “because standard training and evaluation procedures reward guessing over acknowledging uncertainty” — the behaviour is an incentive, not a defect.

Its definition is worth keeping, because it explains why these errors are so hard to spot: hallucinations are “plausible but false statements”. Not gibberish. Exactly the shape a right answer would take.

The exam analogy, which is theirs

Think of a multiple-choice test. You do not know the answer. Guessing might get lucky; leaving it blank guarantees zero.

OpenAI applies this to a model asked for someone’s birthday it does not know. Guess “September 10” and there is a 1-in-365 chance of being right. Say “I don’t know” and the score is certainly zero. Across thousands of questions, “the guessing model ends up looking better on scoreboards than a careful model that admits uncertainty.”

That is the whole mechanism. Nobody trained a model to lie. They graded it on how often it was exactly right, and never on how often it was confidently wrong — and the model learned the obvious lesson.

The table that makes it concrete

This is the part worth carrying away. OpenAI published these figures for two of its own models on the SimpleQA evaluation:

	gpt-5-thinking-mini	OpenAI o4-mini
Abstention (no answer given)	52%	1%
Accuracy (higher is better)	22%	24%
Error (lower is better)	26%	75%

Read the accuracy row alone and the older model wins. Read the error row and it is wrong three times as often. Two points of accuracy cost forty-nine points of error rate.

OpenAI’s own summary of it: “Strategically guessing when uncertain improves accuracy but increases errors and hallucinations.”

And since most leaderboards publish accuracy and not error, the model that bluffs on 99% of questions it cannot answer is the one that looks better.

Why the errors are facts, not typos

There is a second half, about why models get spelling right and birthdays wrong.

Pretraining is next-word prediction over enormous quantities of text, where “the model sees only positive examples of fluent language”. Nothing is labelled false.

Then the analogy that makes it click. Label millions of pet photos “cat” or “dog” and a model learns to classify them. Label each photo with the pet’s birthday and it can never learn, “since birthdays are essentially random”. OpenAI draws the line directly: spelling and parentheses “follow consistent patterns, so errors there disappear with scale. But arbitrary low-frequency facts, like a pet’s birthday, cannot be predicted from patterns alone.”

That is why a model is flawless on grammar and invents a citation. Grammar is a pattern. A specific dissertation title is not — and when OpenAI asked a widely used chatbot for one paper author’s dissertation title, it “confidently produced three different answers—none of them correct.”

Two things people get wrong

“A better model will fix it.” Partly, and only partly. OpenAI’s own finding is that “accuracy will never reach 100% because, regardless of model size, search and reasoning capabilities, some real-world questions are inherently unanswerable.”

“So hallucination is inevitable.” No — and this is the more useful correction. Their finding is explicit: “They are not, because language models can abstain when uncertain.” A model that says it does not know has not hallucinated. What is missing is not capability, it is a scoreboard that rewards saying so.

What to actually do

• Distrust confidence, not fluency. The failure mode is a well-formed, plausible, wrong answer. Fluency tells you nothing about truth, and it never did.
• Ask questions that have a checkable answer, and check it. Names, dates, citations, part numbers, quotations — the arbitrary low-frequency facts — are exactly where the mechanism above predicts failure.
• Treat “I don’t know” as a good sign. Under OpenAI’s own Model Spec it is the intended behaviour: better “to indicate uncertainty or ask for clarification than provide confident information that may be incorrect.”
• Read error rates, not just accuracy, when comparing models. Most leaderboards publish only the metric that rewards guessing. If a comparison does not show what share of answers were confidently wrong, it is not telling you the thing you care about.
• Never pass on a fact you have not verified. That applies to a model’s output the same way it applies to anything else — and if you want the shape of the problem in your own work, whether AI detectors can tell the difference is a related lesson in trusting a confident score.