Start

Home / Big ideas

Idea6-min read+5 pts

How to Test an AI Tool: Ten Questions, Not One

With Josh Schultz ·

Answer. To test an AI tool, ask it 5 to 10 questions you already know the answers to, and score each one. This is a test set (a fixed list of questions with known answers). One good answer proves nothing, but a scored set shows you where the tool slips.

The picture

Diagram: on the left, one question, "Who supplies RL-235?", gets a check mark and the note "looks fine." On the right, ten scored questions show how the tool handles easy lookups, dates and math, and a question that is not in the file. The not-in-the-file answer is circled.

The left side shows how most people test a tool. They ask one question, get one good answer, and decide it works. The right side shows the same tool with ten questions, each one scored, so you can see how it handles easy lookups, dates, and a question whose answer is not in the file. The circled one matters most, because a weak tool tends to make up an answer there instead of admitting it does not know.

Think of it like this

You would never check one part from a lot of 500 and ship the whole lot. You pull a sample, check each part against the print, write down what you found, and then decide whether the lot is good.

A test set works the same way. Your questions are the sample parts, your answer key is the print, and your scorecard is the inspection report. When the setup changes, you check again, just as you would re-run a first article inspection after a tooling change.

Why does it matter at work?

One good answer feels like proof, but it is not, because the trouble shows up later when someone asks a question you did not try. Here is how I put it: "People like to just throw one question like yeah, it works fine, and then they wonder why people complaining a couple days later."

Here is how that plays out. Gloria Benitez runs quality at Mesquite Ridge Metal Works, and she loads the receiving procedures into an AI assistant for the dock. She asks one question about cert checks for bar stock, and the assistant answers it perfectly, so she tells the team it is ready. Two days later, a receiving clerk asks about a partial shipment, and the assistant gives a step that is not in the procedure.

So Gloria writes ten questions she knows cold, taken straight from the binder, and scores the assistant: 7 right, 2 partly right, and 1 wrong. She tightens the instructions and runs the same ten questions again. Because the questions did not change, she knows that any difference in the score came from her changes.

How does it work?

A good test set has four parts.

Loop diagram: write the answer key, run the questions, score each one, change something, then run the same questions again. "Write the answer key" is underlined with the note "before anyone runs it."

The diagram shows the cycle. You write the answer key first, run the questions, score each one, change something, and then run the same questions again.

Questions you know cold. Pull them from real work, and have the person who knows the job write the answer key before anyone runs the test, so the key cannot be shaped by what the AI said.

A mix of easy and hard. Add a math question, a date question, and a question with a fuzzy word like "soon" or "under two weeks." Also add one question whose answer is not in the file at all. A good tool says "not listed," while a weak one makes something up.

One way of scoring. Mark each answer right, partly right, or wrong, and write it down every time. A scorecard you can compare from one run to the next is far more useful than a gut feeling.

Run it again after every change. Whenever you change the instructions, add files, switch tools, or the tool gets an update, run the same set again. The set is your gauge, so you should not change it each time.

Where do people get it wrong?

Most mistakes come from testing to feel good instead of testing to find problems.

  1. One question and done. One right answer only tells you about that one question, so ask more.
  2. Only easy questions. Easy questions tell you very little, so include the dates, the math, and the question whose answer is not in the file.
  3. Grading after reading the answer. AI sounds sure of itself, which makes it easy to be generous, so write the answer key first. See how AI writes for why it sounds so sure.
  4. New questions every time. If the questions change, you cannot tell whether the tool got better or the test got easier.

Try this in 2 minutes

This test uses a small made-up supplier list. Use any chat tool, and paste in the list first.

SAMPLE DATA. FICTIONAL COMPANY. NOT REAL.
Supplier | Parts | Terms | Lead time
Red Creek Steel Supply | MR-3001, MR-3002 | Net 30 | 10 days
Panhandle Bar and Plate | MR-3001, MR-3002 | Net 45 | 21 days
Bolt Barn Industrial | MR-2001, MR-2002 | Net 30 | 5 days, priced per 100
Blue Heron Coatings | MR-2010, RL-250 | Net 15 | 7 days, $250 minimum order
Lone Oak Harness | RL-235 | Net 30 | 14 days
Northbay Circuit Assembly | RL-230 | Net 30 | 21 days
Kestrel Electronics | RL-230, RL-231 | Prepay | 112 days

Use only the supplier list above.

Then ask these questions one at a time, and score each answer against the answer key in the table.

Question Answer key
What does Red Creek Steel charge for MR-3001? Not on the list
We need 10 parts powder coated at Blue Heron for $8 each. What will we pay? $250 (the minimum)
If I order RL-230 from Kestrel today, November 19, what date does it arrive? March 11
If I order MR-3001 from Panhandle today, November 19, what date does it arrive? December 10
Which suppliers can ship in under two weeks? Red Creek, Bolt Barn, Blue Heron

A scorecard spreadsheet with five questions. Each row shows the answer key, a short quote of what the AI said, and a score. All five say right.

When I tried this, I first asked five easy questions, such as who supplies a part and how many suppliers are on Net 30. Every answer matched my key, and if I had stopped there, I would have said the tool works.

Then I asked the five questions above, which are the ones that can trip a tool up, and this time it passed every one. Here is what it said on each.

  • Not in the file. "The supplier list doesn't include pricing, so I can't say what Red Creek Steel Supply charges for MR-3001." It did not make up a price.
  • Minimum order. "You would pay $250, not $80."
  • Dates. "It arrives on Thursday, March 11, 2027." For the Panhandle date it added: "This assumes the lead time is in calendar days. The list doesn't say whether it means business days."
  • A fuzzy word. "Lone Oak Harness is exactly 14 days, so it doesn't qualify as under two weeks."

That is a good tool, and it even showed its guesses, such as assuming calendar days, so now you know what to check. A clean score is still a useful result, because it tells you what you can trust today. If you run the same set again after an update, new files, or new instructions, you will see whether anything has changed.

Frequently asked questions

How many questions do I need?

Start with 5 to 10, which is enough to find the soft spots in an afternoon. Then add a question every time someone finds a new mistake, so the same slip gets caught next time.

Where do the answers come from?

They come from the person who knows the work best, written down before the test runs. Do not use the AI's own answers as the key, because then you are grading it against itself.

Can AI write the test questions for me?

It can suggest questions, and that helps. But a person who knows the job has to write and check the answer key, because the key is the whole point of the test.

Do I need to retest when the tool updates?

Yes, because tools change under you. Rerun the same set after any update, new files, or new instructions, which takes about ten minutes and tells you whether anything moved.

Use it today

From my training: this comes from how I test every AI setup before a team relies on it: a scored set, not one good answer.

Ideas are worth 5 points. Guides are worth 10 and up.

Get your first win this week.

One minute. Three guides picked for your job.

Start my first win