Why AI Makes Things Up: It Can Sound Sure and Still Be Wrong
With Josh Schultz ·
Answer. AI makes things up because it writes the most likely words, and it does not look anything up unless you give it your file or turn on a search. People call this a hallucination. A likely answer sounds as sure as a true one, so the tone tells you nothing. Hand it the source, ask for the quote, and let it say "not stated."
The picture

The top row shows a question that nothing in the chat can answer, so the AI has only its general sense of what usually comes next. The bottom row shows the same question with your file in the chat, where the AI can read the real text and point to the line it used. The circled box is the one you want, because a quote is something you can check, while a smooth answer is something you can only believe.
Think of it like this
You have probably worked with someone who answers every question in a meeting right away, in a steady voice. When they know the answer they are right, and when they do not they sound exactly the same, so you learn to ask, "Where did you see that?" If they open the file and read you the line, you can trust it, and if they answer from memory, you check it before it goes in an email.
AI is that colleague, with one difference, which is that it is not trying to bluff. It writes the words most likely to come next, which is the idea behind how AI writes, and a steady, specific-sounding answer is simply the most likely kind of answer.
Why does it matter at work?
A made-up answer is expensive when it leaves your desk, because other people trust it and build on it, such as a wrong date in a customer email or a wrong clause in a decision.
Here is an example. Dana Ortiz, the senior buyer at Mesquite Ridge Metal Works, asks an AI, "What are the payment terms in our Red Creek Steel Supply agreement?" without pasting the agreement. A tool that answered anyway would probably say "net 30," because that is the most common term and it would sound right. Dana might pass it to Luis Ramos in accounting, who would schedule payments around it, and nobody would notice until the real agreement said something else.
How does it work?
Four ideas explain most made-up answers, and the habits that follow from them are short.
It writes likely words, and likely is not the same as true. The AI never checks a fact before it writes it. It writes what usually follows, so a common-sounding answer comes easily even when it is wrong for your situation.
It only looks things up when you give it something to look at. If your file is in the chat, or your tool can search the web or read your files, the AI can answer from that text. If not, it is working from what it learned during training, and that does not include your contracts, your prices, or last week's email. The lesson on what a context window is shows what the AI can and cannot see.
Tone does not tell you which case you are in. A smooth sentence with a specific number reads as checked, whether or not it was. Newer tools say "I can't verify that" more often, and the one I tested did, but that varies by tool and by question.
Three habits catch most of it. Paste the source instead of naming it, ask for the exact quote the answer came from, and tell the AI it may reply "Not stated" when the text does not say. Together they turn a guess into something you can check.

The picture lists the three habits in the order you would use them. The third one matters most, because without a way out the AI feels it has to answer, and the easiest answer to write is the likely one.
Where do people get it wrong?
Most trouble comes from trusting the tone, or from checking in a way that does not really check.
- They trust a specific detail because it is specific. Part numbers, dates, page numbers, clause numbers, and quotes are the details AI is most likely to get slightly wrong, because they look so exact. If you did not give it that detail, check it.
- They ask "Are you sure?" The AI may just agree with you or change its answer, which tells you nothing about the facts. Ask for the quote from your file instead.
- They never give it a way out. A question like "What are the payment terms?" invites an answer, while "What are the payment terms? Reply 'Not stated' if the text does not say" invites an honest one.
- They check only the answers that look strange. The made-up answers that cause trouble look normal, so pick one answer in each batch and look it up in the source.
Two signs should slow you down: a detail you never supplied, such as a date or a name, and a different answer when you ask the same question in a new chat.
Try this in 2 minutes
This test shows what your AI tool does when it has nothing to go on, and what changes when it has a file. Use any chat tool, and open a new chat for each step.
- Paste this question about an article I made up, including the title, the authors, and the journal:
Summarize the main conclusion of the 2019 article "Shop Floor Variance Costing" by Mendez and Hartley in the Journal of Manufacturing Finance.
- Open a new chat and paste this short agreement and question. The agreement says nothing about payment, on purpose:
Use only the text below to answer. Quote the sentence you used. If the text does not say, reply "Not stated."
Text:
SUPPLY AGREEMENT: Red Creek Steel Supply and Mesquite Ridge Metal Works
1. Pricing. Prices for 4140 round bar are fixed for 90 days from the date of each quote.
2. Delivery. Red Creek will deliver within 10 business days of a confirmed purchase order. Freight is billed at cost.
3. Returns. Material that fails the agreed mill certificate may be returned within 30 days.
Question: What are the payment terms?
- Here is what I hope you see: in step 1, the tool admits it cannot find the article, and in step 2 it answers "Not stated." If step 1 gives you a confident summary, you have just seen a made-up answer, so be careful with that tool. To see a quote, change the question to "How long are prices fixed?" and run step 2 again.

When I tried this, the tool did the right thing both times. For the made-up article, it said: "I can't verify that this article exists, and I don't have reliable knowledge of 'Shop Floor Variance Costing' by Mendez and Hartley (2019) in the Journal of Manufacturing Finance. If I summarized its conclusion, I'd be making it up, and a plausible-sounding fabrication is worse than no answer here." For the agreement that never mentions payment, the reply was one line: "Not stated."
I also asked for a supplier lead time it had no way to know, in one line with no caveats, and it still said it did not have that information. That is encouraging, but tools differ, and the same tool can behave differently on a question that sounds more ordinary than an invented article, so the habits in this lesson are worth keeping either way.
Frequently asked questions
What is an AI hallucination?
It is an answer that sounds right but is made up, such as a date, a quote, or a source that does not exist. The AI is not lying on purpose. It wrote the most likely words, and they were wrong.
Does AI still make things up with newer tools?
Yes, though some tools do it less often. It is most likely with details the AI half-knows, such as names, numbers, page numbers, and quotes, and with anything about your own files.
How do I ask AI for sources?
Paste the source into the chat, then ask: "Quote the sentence you used and tell me where it is." If your tool can search the web, ask for the link and open it yourself.
What should I say so it does not guess?
Add one line: "If the text does not say, reply 'Not stated.'" It gives the AI an honest answer to use when the information is missing.
Use it today
- Email suppliers about what's missing from their quotes: the same "not stated" idea, used to find what a quote leaves out.
- Related ideas: How AI writes: one likely word at a time and AI limitations: the three gaps.
From my training: this lesson builds on my field notes "AI Exaggerates" and "Best, Not Predictable," which are also the source for how AI writes.