Start

Home / Big ideas

Idea6-min read+5 pts

How AI Answers From Your Files: What Chat With Your Documents Means

With Josh Schultz ·

Answer. When you chat with your documents, many tools do not read every page each time. They first search your files for the parts that match your question, and then the AI answers from only those parts. This step is called retrieval, which means fetching, and because the search can miss a part, you should ask for the quote and the page number so you can check.

The picture

Diagram: a folder labeled "Your files" goes into a box that says "Search for the parts that match your question." Three pages come out: page 3, page 7, and page 12. They go to a box that says "AI answers from those parts," which produces "Answer plus page 3." The search box is circled, with a note that says "it can miss the right page here."

Follow the picture from left to right. Your files go into a search step, which pulls out the few pages that look most like your question, here pages 3, 7, and 12. The AI reads only those three pages and writes an answer that points back to page 3. The search box is circled because that is where a quiet wrong turn can happen, since the AI never sees the pages the search left behind.

Think of it like this

Imagine you ask a file clerk a question about a contract, and the cabinet holds 200 folders. A good clerk does not re-read all 200. They pull the three or four folders that look right, flip to the likely pages, and answer from those, which works well most of the time. It also means the answer is only as good as the folders the clerk pulled, and if they grabbed the wrong one, they would answer from the wrong pages in the same calm voice.

That clerk is what "chat with your documents" means. The AI does the answering, but a search step does the pulling first, and a smooth answer can come from a pull that missed the best page, which ties back to why AI makes things up.

Why does it matter at work?

When the answer sits where the search did not look, you get a confident answer that is incomplete, and you may never know it.

Here is an example. Dana Ortiz, the senior buyer at Mesquite Ridge Metal Works, loads 40 supplier agreements into a document tool and asks, "Which suppliers charge a restocking fee?" The tool searches, pulls the pages that look most like her question, and answers from those. If one agreement calls it a "return processing charge," the words do not match hers, so that page may not be pulled at all. And since she asked about all 40 agreements, the tool may stop after a handful of matches, so the answer reads like a complete list when it is not.

How does it work?

There are four ideas behind most of these tools, and each one explains something you can see on screen.

Your files are cut into pieces and indexed. The tool splits each file into small pieces, usually a few paragraphs, and keeps a searchable record of them. When your files are short, many tools skip this and put the whole file in front of the AI at once.

Only the best-matching pieces go to the AI. The AI can only read so much at one time, which is the limit described in what a context window is, so the search picks the pieces most likely to hold your answer. Everything else stays out of sight, even if it matters.

Good tools show where an answer came from. Tools built around your own files, such as Google's NotebookLM, which Google's help pages now call Gemini Notebook, attach citations to the answer. According to Google, you can hover over a citation to see the quoted text, or select it to jump to the passage in your source. Other chat tools vary, so it is worth asking each one to cite.

You can ask for page numbers yourself. A line such as "Give the page number and quote the sentence you used" makes the AI point to its evidence. Page counts can differ from the number printed on the page, because the tool may count from the start of the file, so open the page and look.

Diagram: three columns, each with a way the search misses and a fix. "You use different words than the file" is fixed by "Ask in the file's own words." "The answer is spread across many pages" is fixed by "Name the section or page to read." "You ask for every item or a count" is fixed by "Ask for a list, then check it." The last fix is circled. A note says "Always ask for the page number, then look at it."

The second picture shows the three most common misses and what to change in how you ask.

Where do people get it wrong?

Most mistakes come from assuming the AI read everything, when it may have read only what the search handed it.

  1. They ask "every" or "how many." A question about all the files, or a count, needs every piece to be read, and the search only returns the top few. Ask for a list instead, then check it against the files.
  2. They use their own words, not the file's. The search matches meaning and wording, so "restocking fee" may not find "return processing charge." If the first answer looks thin, try again with the words the file uses.
  3. They trust a citation without opening it. A citation points to a passage that may only loosely support the claim, so click it and read the line.
  4. They load scans and expect them to be searched. A scanned page is a picture, and a tool that cannot read the picture has nothing to search. The lesson on what AI can read in PDFs and images explains the difference.

Try this in 2 minutes

This test shows the answer, the page number, and the quote you should expect from any tool that works from your files. It pastes a short five-page agreement into the chat with page marks. With your own file, attach it to the chat instead.

  1. Open a new chat and paste this:
Use only the document below. Answer the question, give the page number, and quote the sentence you used. If the document does not say, reply "Not stated."

Document:
[Page 1] SUPPLY AGREEMENT between Red Creek Steel Supply and Mesquite Ridge Metal Works. This agreement covers 4140 and 1018 bar stock. Orders are placed by purchase order.
[Page 2] Pricing. Prices are fixed for 90 days from the date of each quote. A quote may be extended once by written request.
[Page 3] Delivery. Red Creek delivers within 10 business days of a confirmed purchase order. Freight is billed at cost. Late deliveries of more than 5 days earn a 2 percent credit on that order.
[Page 4] Returns. Material that fails the agreed mill certificate may be returned within 30 days. Return freight is paid by Red Creek.
[Page 5] Payment. Invoices are due net 45 days from the date of receipt. Early payment within 10 days earns a 1 percent discount.

Question: What credit do we get if a delivery is late?
  1. Check the page number and the quote against the text. Then change the question to "What happens if Red Creek raises its prices during the contract?" and run it again. The agreement never says, so a careful tool should say so.

A chat with the five-page agreement and the question about late deliveries. The reply has three labeled lines: Answer, "A 2 percent credit on that order, applying when a delivery is more than 5 days late"; Page, 3; and Quote, the sentence from page 3. The page number is circled.

When I tried this, the reply came back in three labeled parts. The answer was "A 2 percent credit on that order, applying when a delivery is more than 5 days late." The page was 3, and the quote was "Late deliveries of more than 5 days earn a 2 percent credit on that order." I could look at page 3 and see the same sentence, which is the check you want to be able to make.

For the second question, the reply began with "Not stated." and then explained: "The closest provision is on Page 2: 'Prices are fixed for 90 days from the date of each quote.' That covers how long a quoted price holds, but it doesn't address price increases during the contract or any consequence for them." It pointed to the nearest text without pretending it was an answer.

Because the whole agreement was pasted into the chat, nothing was lost to a search step. With a large pile of files the search comes first, which is why the page number and the quote matter even more.

Frequently asked questions

What does chat with your documents mean?

It means you load files into an AI tool and ask questions about them. The tool searches your files for the parts that match, and the AI answers from those parts, so the answer comes from your material instead of general knowledge.

Why does the AI miss things that are in my files?

The search may not pull the right page, because your words differ from the file's, the answer is spread across many pages, or the question needs every item.

Does the AI read my whole file?

Sometimes. A short file may be given to the AI whole, while a large file or a big pile of files is searched first. Test it with a fact you know is on a late page.

Is it safe to upload my files?

That depends on the tool and your plan, so check your plan's data settings before you upload anything real. Business plans usually treat uploaded files differently from free plans. Our safe use page lists what to check.

Use it today

From my training: this idea sits behind every guide where AI reads your PDFs, so the page-number habit carries over to all of them.

Ideas are worth 5 points. Guides are worth 10 and up.

Get your first win this week.

One minute. Three guides picked for your job.

Start my first win