Start

Home / Big ideas

Idea6-min read+5 pts

Show the Evidence Rows Behind Every AI Count

With Josh Schultz ·

Answer. To show the evidence rows behind an AI count, ask the AI to keep, with every number or claim, the rows it came from and the exact words that made it count them. Evidence rows are the lines of your own data that sit behind a number. With them, anyone can check a claim in about a minute instead of taking it on trust.

The picture

Diagram: a claim, "PB-02: 3 leaks," points to three work orders, each with the exact words from its note: WO-5201 "oil on floor under brake 2," WO-5202 "hyd hose weeping at left cylinder," and WO-5205 "oil puddle again." A note underneath says "the rows and the exact words."

On the left is the claim, "PB-02: 3 leaks." On the right are the three work orders that make it true, each with the words someone wrote on it. A number alone is just an opinion in a nice font. The same number with its rows next to it is something you can open, read, and agree with or push back on.

Think of it like this

Think of an expense report. Each line says something like "Travel, $412," and by itself that is only a claim. The reason finance staff accept it is the receipt stapled behind the line, which shows the date, the amount, and the vendor. An auditor does not re-do your whole trip. She picks a line, flips to the receipt, and sees in thirty seconds whether the two agree.

An AI count should work the same way. The number is the line on the report, and the evidence rows are the receipts stapled behind it.

Why does it matter at work?

It matters because AI can give a confident number that is wrong, and a bare number gives you nothing to check it with. When a count comes with its rows, a mistake shows up in a minute, and a good count wins the argument without anyone raising their voice.

At Mesquite Ridge Metal Works, "press brake 2 failed five times" starts an argument, because everyone remembers the machine differently. Five work order numbers, each with the words from its notes, end it. The plant manager can read the five rows and see the pattern, such as the same cylinder fitting coming loose again, which a bare count would never show. The invoice check works the same way: every flag lists the invoice, the line, what was billed, and what the PO says, so the buyer can look at one line and know whether to make the call. That is also why the repeat failures guide has the AI read and a script count, since AI can count the same rows two ways on two days.

How does it work?

There are four habits. You ask for the row ID, you ask for the exact words, you ask what was left out, and you check one row yourself. Each one costs a few extra words in the prompt.

Ask for the row ID with every claim. That means the work order number, the invoice number and line, or the row number in your sheet, anything that lets a person find the original in seconds.

Ask for the exact words. Say "copy the words exactly," because a summary is the AI's opinion of the note, while the exact words are the note. When the words are copied, a person can search for them in the original.

Ask what was left out and why. A count with only the rows that counted hides every judgment about the rows that did not. A short "looked at and left out" list shows the borderline calls, such as a monthly PM or a duplicate.

Check one row yourself. Open the original, search for the words, and see if they say what the AI says they say. Anyone can do this, and nobody has to trust the tool.

Diagram: three boxes in a row. 1, open the work order, for example WO-5202. 2, search for the words with Ctrl+F. 3, do they say what AI says, yes or no. The last box is circled, with a note that says if the answer is no, the count is wrong.

This is an audit trail for a single number, and it is the same habit that testing any AI tool depends on, since you can only test an answer you can see the basis for.

Where do people get it wrong?

Most mistakes come from collecting evidence that nobody looks at, or that cannot be checked.

  1. The "evidence" is a paraphrase. If the AI rewrites the note in its own words, you cannot search for it, so tell it to copy the words and check that at least one of them appears in the original.
  2. Only the winners get evidence. Ask for the rows that were left out as well, because that is where the judgment calls hide.
  3. Nobody opens a row. Rows no one reads are only decoration, so pick two or three at random every time and check them.
  4. The evidence shows a rule you did not mean. When the rows are in front of you, you may find the AI counted something you would not have, and that is useful, because now you can fix the rule.

Try this in 2 minutes

This test gives AI eight made-up work orders and asks which machine had the most hydraulic oil leaks. You will ask once with no evidence and once with it. Any chat tool works.

  1. Open a new chat and paste this:
Here are 8 work orders.
WO-5201, PB-02: oil on floor under brake 2, tightened fitting
WO-5202, PB-02: hyd hose weeping at left cylinder, retorqued
WO-5203, MILL-07: coolant pump no flow, chip screen packed
WO-5204, PB-02: PM done, no leaks found
WO-5205, PB-02: oil puddle again, replaced seal kit
WO-5206, LASER-01: chiller alarm, topped off. Mill 7 low on coolant too
WO-5207, MILL-07: coolant leak at hose clamp
WO-5208, PB-01: hydraulic oil weeping at pump
Which machine had the most hydraulic oil leaks? Answer in one line.
  1. Open a second new chat and paste the same eight work orders with this question at the end:
Which machine had the most hydraulic oil leaks? Give the count. Then list every work order you counted as a table with three columns: work order, machine, and the exact words from its note that made you count it. Copy the words exactly. Also list any work order you looked at and left out, with the reason.
  1. Here is what I expect: PB-02 with 3 leaks, from WO-5201, WO-5202, and WO-5205. WO-5204 should be left out because it is a PM with no leak, and WO-5207 because it is a coolant leak.
  2. Pick one row from the table, find its words in the list you pasted, and use search to confirm they are really there.

A chat reply: PB-02 had 3 leaks and PB-01 had 1. A table lists three PB-02 work orders with the exact words from each note, which are circled, plus a PB-01 row. Below it is a list of work orders left out and why.

When I asked for one line, the answer was right: "PB-02 had the most hydraulic oil leaks, with 3 work orders (WO-5201, 5202, 5205)." But it gave me nothing to check, and it said nothing about how it decided. With the evidence request, the reply first put the PB-02 count at 4, then stopped itself with "Wait, WO-5208 is PB-01, not PB-02" and fixed the count to 3. The table made that slip easy to see. The reply also added a caveat: "WO-5201 ('oil on floor under brake 2') and WO-5205 ('oil puddle again') don't say 'hydraulic' explicitly." That is a judgment call, and the one-line answer had hidden it.

Frequently asked questions

What are evidence rows?

They are the lines of your own data that sit behind a number or a claim, such as the work orders behind a repeat failure count. Keeping them with the number lets anyone check the claim against the original.

Can AI get the exact words wrong?

Yes, it can drop or change a word, so search for the words in the original before you trust them. In the guides on this site, a script writes the evidence rows from the data, which removes that risk.

How many rows should I check?

Check two or three, picked at random, plus every row the AI flagged as unsure. If one of them is wrong, check more and fix the rule before you use the count.

Does this work for money numbers too?

Yes, and it matters more there. An overbilled invoice flag should show the invoice, the line, the billed amount, and the PO amount, so the buyer can look at one line and decide.

Use it today

From my training: keeping the evidence with every count is the fourth of five methods in my repeat failures workflow, so each number points back to the work orders behind it.

Ideas are worth 5 points. Guides are worth 10 and up.

Get your first win this week.

One minute. Three guides picked for your job.

Start my first win