Start

Home / Big ideas

Idea6-min read+5 pts

Check What a Fix Changed: Rerun Every Test After Every Fix

With Josh Schultz ·

Answer. After every fix, rerun every test and compare the results before and after. A fix can repair one problem and quietly break something that passed before, and people call that a regression, which means something that worked has gone backward. If you check only the test that failed, you will miss it.

The picture

Diagram: a grid with four contract lines down the side and two columns, version 1 and version 2. Version 1 has two crosses, on lines 3 and 4. Version 2 fixes those, but has two new crosses on lines 1 and 2, and those two new crosses are circled.

The grid shows the same four checks run on the old version and the new version of a skill. In the first column, two lines fail, and the fix was meant to repair exactly those two. In the second column they pass, which looks like success, but two different lines now fail, and those two used to pass. If you only looked at the lines you meant to fix, you would not see the circled ones.

Think of it like this

Picture a machinist who adjusts a cutter to bring one diameter into tolerance. A good machinist does not measure only that diameter and walk away. They run the first part again against the whole print, because a change to one setting can move a different dimension.

The change was right for the dimension they were chasing. The only way to learn what it did elsewhere is to measure everything again, and the part you check last is often the one that moved.

Fixing a skill works the same way. You change one rule to fix one miss, and then you run the whole print again, which means every test you have, so you can see what else moved.

Why does it matter at work?

You cannot see a regression by looking at the fix, because the fix worked. You find it only by rerunning the tests that already passed and comparing, which is what turns a hidden break into a visible one.

Here is what happened in the tuning guide. At Mesquite Ridge Metal Works (a made-up company in our sample data), the shift handoff skill was tested on Tuesday's notes and Wednesday's notes. Version 1 failed three lines on Wednesday, and version 2 fixed all three. On Tuesday, version 2 passed every line.

If I had checked only the three lines that had failed, version 2 would have looked finished. But when I reran the full Wednesday checklist, two lines that version 1 had passed now failed. The title said "Plant not stated," and the question about "RL-321 display" had disappeared. The fix had removed the Monday example's open question about RL-231, which seems to be what taught the skill to compare part numbers, and no rule said the title always says Plant 1.

Version 3 fixed those two, and both tests passed every line. The skill had said "every line passes" at version 2, too, so a person doing the comparison is what caught it (make AI check its own work).

How does it work?

The habit has four parts. You keep your tests, rerun all of them, compare before and after, and make smaller fixes when something breaks.

Keep every test. A test is one real input with a checklist, such as Tuesday's notes with the Tuesday checklist. Each test runs in its own new chat, so an old chat cannot carry the corrections (how to test an AI tool).

Rerun all of them after every fix, not only the one that failed. This is the whole idea. A test that passed before is the test most likely to tell you what the fix moved.

Compare before and after for the same input. Save the old result and the new result as two files, and ask Claude to compare them line by line against the contract. You are looking for lines that went from pass to fail, which are the regressions, and lines that went from fail to pass, which are the fixes.

Fix what broke with a small change. If a fix broke something, change one or two things at a time, so you can tell which change caused what (change one thing at a time). When a change makes things worse, go back to the last version that passed (save every working version).

Diagram: one fix leads to two boxes, the Tuesday test and the Wednesday test, each in its own new chat. Both lead to a box that says compare with before, which is circled. From there, one arrow says lost a line, fix it again with a smaller change, and the other says nothing lost, save the version.

Where do people get it wrong?

Most mistakes come from checking only what you meant to change. These are the four I would watch for.

  1. Rerunning only the failed test. It passes, so you stop, and the line you broke on a different test goes unseen.
  2. Trusting the skill's own report. It may say every line passes while two do not, so you walk the checklist yourself.
  3. Making many fixes at once. When something breaks, you cannot tell which of the five changes did it.
  4. Comparing from memory. "I think it was fine before" is not a comparison. Keep the old result so you can put the two side by side.

Try this in 2 minutes

This test gives Claude a short contract and two handoffs, the old one and the new one, and asks what the fix changed. It is a shortened version of the two Wednesday handoffs in the guide. You can use any chat tool.

  1. Open a new chat and paste this:
Here is a contract for a shift handoff:
1. The title begins "Plant 1 Shift Handoff".
2. "RL-321 display" is kept as written, and there is a line under Open questions that says our display is RL-231.
3. Every line under "Needs a decision" starts with "Owner:".
4. Open questions does not ask which job the 2 rewelded frames belong to.

Below are two handoffs written from the same notes. BEFORE is from the old version of my skill. AFTER is from the new version, after I fixed some problems. For each contract line, say Pass or Fail for BEFORE and for AFTER, in a table. Then list what got better and what got worse.

BEFORE:
Plant 1 Shift Handoff: 2nd shift, Wed 11/18
Needs a decision
- Ray: give the ok on the Saturday overtime.
Open questions
- It is unclear which job the 2 rewelded frames belong to.
- The notes say "RL-321 display." Our display is RL-231, so it is unclear which is meant.

AFTER:
Plant not stated Shift Handoff: 2nd shift, Wed 11/18
Needs a decision
- Owner: Ray. Give the ok on the Saturday overtime.
Open questions
- The 1015 shafts count is 41 on the cart and 42 on the tag.
  1. Read the table. Count the lines that went from Pass to Fail.
  2. Try it with your own before and after results.

A chat reply with a table of four contract lines for BEFORE and AFTER. Lines 1 and 2 go from Pass to Fail, and those two Fail marks are circled. Lines 3 and 4 go from Fail to Pass. Under "What got worse," the lost Plant 1 title and the dropped RL-231 question are listed.

When I tried this, the table showed two lines getting better and two getting worse. Under what got better, Claude wrote "The decision item is now 'Owner: Ray. Give the ok on the Saturday overtime.'" and "The rewelded-frames question is gone from Open questions." Under what got worse, it wrote "The title now reads 'Plant not stated Shift Handoff'" and "The 'RL-321 display' quote and the RL-231 clarification are both missing."

It ended its summary with "Each version passes 2 of 4 lines, so the fixes traded one set of failures for another." That is the whole lesson in a sentence. It also guessed at the cause, that the new version "may be over-pruning open questions to satisfy line 4," which is a useful lead for the next small fix.

Frequently asked questions

What is a regression?

It is something that used to work and no longer does, because of a change somewhere else. In a skill, it is a checklist line that passed on the old version and fails on the new one.

Do I really have to rerun every test every time?

Yes, because the one you skip is the one that may have moved. With two or three tests, it takes a few minutes, and it is much quicker than finding the break after the skill is in use.

How many tests do I need?

Start with two or three real inputs that look different from each other. When real use turns up a new kind of miss, save that input as another test and rerun it with the rest.

What if the fix made things worse overall?

Go back to the last version that passed, then try a smaller change. Keeping every version makes that a one-minute step.

Use it today

From my training: rerunning every test after every change is a habit from how we tune the skills we build for client work.

Ideas are worth 5 points. Guides are worth 10 and up.

Get your first win this week.

One minute. Three guides picked for your job.

Start my first win