Start

Home / Big ideas

Idea6-min read+5 pts

What Are AI Evals? The Test You Save and Rerun Every Time Something Changes

With Josh Schultz ·

Answer. An eval (short for evaluation) is a saved set of test questions with known right answers that you run again every time something about your AI setup changes. A script can often score the answers for you, so in a few seconds you learn whether the change made things better, worse, or no different.

The picture

Diagram: a clipboard labeled "Saved test set, 10 questions with answers" feeds a box labeled "Run the AI," which feeds a box labeled "Score." The score box shows "10 out of 10." A curved arrow runs from the score back to the start with the note "after every change."

Follow the picture from left to right. The saved test set goes into the AI, and the AI's answers go to a scoring step that produces a number, here 10 out of 10. The curved arrow along the bottom is the part that makes it an eval, because it says you do the whole loop again after every change. The note along the bottom is the habit that matters, since the score only means something when you can compare it with the last one.

Think of it like this

When a shop changes tooling, the quality team does not simply hope the parts are still good. They run a first article inspection, which means measuring the first part off the new setup against the print before the rest of the run goes forward. And before anyone trusts a caliper, they check it against a gauge block that has a known size.

An eval is both of those things, kept on a shelf and used again. The test questions are the part to be measured, the answer key is the print, and the script that scores the answers is the gauge. The important word is "again," because a one-time check tells you how things were on one day, while a saved check tells you what changed.

Why does it matter at work?

AI setups change often, and any change can quietly alter how the AI behaves, so a saved test tells you at once whether something broke.

You reword an instruction, add a file, switch to a newer model, or your company turns on a different tool. Most of the time the change helps, but now and then it breaks something that used to work, and you will not find out until a coworker gets a wrong answer.

Here is an example. Gloria Benitez runs quality at Mesquite Ridge Metal Works, and her team uses an AI helper to tag incoming supplier emails as quotes, invoices, delays, or other, so the right person sees each one. She wrote ten real subject lines and the correct tag for each, and she saved them in a spreadsheet. When someone suggested adding one sentence to the instructions to make the tool "more careful," she ran the ten lines again before rolling it out. The score dropped, so the sentence stayed out, and it took her about a minute to find that out instead of a week of misfiled emails.

Diagram: two score bars. The original instructions score 10 out of 10, and the instructions with one added sentence score 6 out of 10. The four empty boxes in the second bar are circled.

The second picture shows what she saw. Each box is one test question, a filled box is a right answer, and the four empty boxes at the end of the lower row are the questions the added sentence broke.

If you are the buyer, the planner, or the person who signs off on a tool, the question to ask a vendor or your IT team is simple: "What do you rerun when you change something, and what was the score before and after?" A good answer names a saved test. A weak answer sounds like "we tried it and it looked fine." For the first step of building such a test, see how to test an AI tool, which explains how to write the ten questions and the answer key.

How does it work?

An eval has three parts, and none of them needs code from you.

A saved set of questions with known answers. These come from real work, and a person who knows the job writes the answer key before anyone runs the test. The how-to lesson above covers how to build it, and this lesson is about the habit of keeping it and using it again.

A way to score each answer. The best scoring is a script, which is a short list of instructions a computer follows the same way every time, such as "compare the tag to the key and count the matches." This works when the answer is a tag, a number, or a part number, which is one reason it helps to ask AI for structured output that is easy to check. Because the script gives the same score for the same answers, it is what people call deterministic. When an answer is a paragraph, a person or a second AI can score it with a short rubric, though that is less steady than a script.

A rerun after every change. Whenever the instructions, the files, the model, or the tool changes, you run the same questions again and write down the score next to the date. Change one thing at a time, as change one thing at a time explains, so that when the score moves you know which change caused it.

Where do people get it wrong?

Most trouble comes from treating the test as a one-time event, or from letting it drift.

  1. They test once and never again. A test you run only at launch tells you nothing about the change you make in March.
  2. They change the questions to fit the AI. If a question is too hard, the temptation is to remove it, but then your score goes up without the tool getting any better. Add new questions when you find new failures, and leave the old ones alone.
  3. They use too few questions, or only easy ones. Include a few that are tricky or ambiguous, and one whose answer is not in the data at all, because that is where weak tools slip.
  4. They trust a score without reading the misses. A score tells you how many were wrong, but only reading the wrong ones tells you why.

Try this in 2 minutes

This test shows an eval doing its job. You will tag ten supplier email subject lines with the same instructions twice, with one small difference, and score both against the answer key.

  1. Open a new chat in any AI tool and paste this:
Tag each supplier email subject line as QUOTE, INVOICE, DELAY, or OTHER. Reply with the numbered list only, one tag per line.
1. Re: Quote Q-2291 for 304 stainless sheet
2. Invoice 88412 is past due
3. Your PO 45017 will ship two weeks late
4. Updated pricing for next quarter attached
5. Thank you for visiting our booth
6. Revised delivery date for bar stock
7. Statement of account for October
8. Lead time change on bearing order
9. Please confirm the new price before we ship
10. Credit memo for returned washers
  1. Here is the answer key: 1 QUOTE, 2 INVOICE, 3 DELAY, 4 QUOTE, 5 OTHER, 6 DELAY, 7 INVOICE, 8 DELAY, 9 QUOTE, 10 INVOICE. Count how many tags match.
  2. Start a new chat and paste the same text, but add one sentence after the first line: "Use OTHER whenever you are not completely sure." Score it against the same key.

A chat window. The user's message is the tagging request with the extra sentence about OTHER. The AI replies with a numbered list in which lines 4, 7, 9, and 10 say OTHER. Those four lines are circled and a note says "6 out of 10."

When I tried this, the first wording matched the key on all ten lines. After I added the extra sentence, the reply tagged lines 4, 7, 9, and 10 as "OTHER," which dropped the score to 6 out of 10. The new sentence sounded careful and reasonable, but it told the AI to give up on every line that was not obvious. Without the saved questions and the key, I would have read a tidy list and moved on. With them, I saw the drop in under a minute, which is the entire point of an eval.

Frequently asked questions

What is an eval in AI?

It is a saved test with known right answers that you run against an AI setup. You rerun it after each change, so you can see whether the change helped or hurt.

How many questions do I need?

Ten is a good start for a small tool, as long as they come from real work and include a few hard ones. Add more each time you find a new failure.

Is an eval the same as testing?

It is testing that you keep and repeat. How to test an AI tool shows how to build the first test set, and an eval is what that set becomes once you rerun it on a schedule.

Can AI score its own answers?

It can score open answers against a rubric, but a script is steadier whenever the answer is a tag, a number, or a part number. Have a person check a few of the AI's scores.

Who should own the test set?

The person who knows the right answers, usually the one who does the job today. Keep it in a shared file, with the date and score of each run.

Use it today

From my training: we teach people to keep a small test with an answer key next to every AI setup they build, and to run it again before they trust any change.

Ideas are worth 5 points. Guides are worth 10 and up.

Get your first win this week.

One minute. Three guides picked for your job.

Start my first win