Human in the Loop AI, Explained: Let AI Clear the Easy 80 Percent
With Josh Schultz ·
Answer. Human in the loop means you let AI and a few simple rules handle the routine work, and you only look at the items that need a person. You decide ahead of time what counts as routine. People call this managing by exception, because your attention goes to the exceptions instead of to everything.
The picture

Ten invoices go into the box on the left, and your rule sorts them into three piles. Six match the quote, so they go through without you. Two are a little off, so you take a quick look. Two have a real problem, so you decide what to do. The circled corner is where your time goes now, instead of being spread across all ten.
Think of it like this
If you have worked on a shop floor, you have probably seen a job status board. Green means the job is running fine. Yellow means someone should take a look. Red means stop and fix it.
A good supervisor doesn't walk to every machine every hour. They watch the board and go to the yellow and red jobs, because that is where their help makes a difference. That isn't cutting corners. It is how one person keeps a whole floor moving.
AI lets you build the same kind of board for paperwork. The routine items show green and move along on their own, and you spend your time on the yellow and red.
Why does it matter at work?
Most office work has two bad options. You can check every line by hand, which takes hours and wears you down, or you can skip the check, which isn't safe. A clear rule plus AI gives you a third option.
Here is an example. Jen Nakamura handles freight at Mesquite Ridge Metal Works. Every week she compares about 60 invoices from Delta Freight Lines against the quotes, line by line, and almost all of them match.
So she writes down three plain rules. An invoice is clear if the billed amount matches the quote. It needs a check if it is up to $25 over. It is a stop if it is more than $25 over, has a charge that was never quoted, or repeats an invoice number. With those rules in place, the matching invoices go through on their own, and she reads about 10 a week instead of 60. Those 10 are the ones that matter, because that is where the extra detention charge and the double bill were hiding.
How does it work?
There are four parts. You write the rule, the tools do the sorting, a person handles what is left, and you count what comes back. As I put it: "The boring 80% becomes a workflow; the 20% that needs judgment escalates back."
You write the rule, not the AI. If you tell the AI to "flag anything weird," you are letting it decide what weird means, and it may decide differently tomorrow. A rule like "stop if billed is more than $25 over quoted" leaves nothing to guess. When the rule is written down in plain words, you get the same sort every time.
The tools do the sorting. Some checks are simple math, like comparing billed to quoted, and a spreadsheet formula can do those (see when not to use AI). AI is most useful for the messy parts, such as reading an invoice that came in as a PDF or noticing a charge that is written out in words.
A person owns the yellow and red. Yellow items get a quick look, and red items get a real decision. Anything that leaves the building still needs a person's OK. When I set these up for a company, the rule is simple: "It will never read something and send it. Everything needs approval."
You count what comes back. If ten lines go in, ten lines should come out.

A tool can drop a line without telling you, and the table will still look complete. Counting the rows is the quickest way to catch that.
Once this is set up, your day changes. In my words: "your job moves to managing by exception versus all this action doing."
Where do people get it wrong?
Most mistakes come from trusting the green pile without knowing how it got there. These are the four I see most often.
- There is no written rule. If the rule only lives in your head, the AI has to guess at it. Write it down in three lines before you start.
- Nobody spot-checks the green pile. Each week, pull two or three green items and check them by hand. That is how you find out if a rule is too loose.
- Nobody counts lines in and out. It takes ten seconds, and it is the only way to notice a line that went missing.
- Red items go out on their own. Red means a person makes the call, every time.
Try this in 2 minutes
This test asks AI to sort ten freight invoice lines using the three rules from Jen's example. You can use any chat tool.
- Open a new chat and paste this:
Sort these freight invoice lines into CLEAR, CHECK, or STOP.
CLEAR: billed matches quoted and there are no extra charges.
CHECK: billed is up to $25 over quoted.
STOP: billed is more than $25 over quoted, OR there is a charge not on the quote, OR the invoice number repeats.
invoice,quoted,billed,extra charges
DF-5101,412.00,412.00,none
DF-5102,386.50,386.50,none
DF-5103,520.00,538.00,none
DF-5104,298.00,298.00,none
DF-5105,645.00,720.00,detention $75
DF-5106,410.00,410.00,none
DF-5107,355.00,367.50,none
DF-5102,386.50,386.50,none
DF-5108,480.00,480.00,none
DF-5109,299.00,299.00,none
Give a short table: invoice, tag, reason. Then one line with the count of each tag.
- Here is what I expect: 6 CLEAR, 2 CHECK, and 2 STOP. DF-5103 and DF-5107 should be CHECK, because they are a little over. DF-5105 should be STOP because of the detention charge, and so should the second DF-5102, because it repeats.
- Count the rows in its table to make sure there are 10. Then read its count line and compare it to mine.

When I tried this, all ten lines came back, so nothing was dropped. But the count line read "CLEAR 5, CHECK 2, STOP 3," which didn't match what I expected. The table showed why. It had marked both DF-5102 lines as STOP, with the reason "Invoice number repeats."
That is a fair reading of my rule, even though I only meant the second one. The count line showed me the gap right away. If I had only skimmed the ten rows, I would have missed it. So count the rows, then compare the totals to what you expected, and when they don't match, go back and read the table.
Frequently asked questions
What does human in the loop mean?
It means a person stays involved at the points that matter. AI and simple rules do the routine work, and a person checks the exceptions and approves anything important before it goes out.
Is managing by exception safe?
Yes, as long as you write a clear rule, spot-check the green pile, and count what comes back. Checking 60 items by hand is actually riskier, because most people get tired and start skimming by item 40.
What should go to a person every time?
Anything that spends real money, goes to a customer or supplier, or can't be undone. Those items stay yellow or red no matter what the rule says.
How do I pick the rule?
Start with how you already decide. Write down when you would sign without looking, when you would look twice, and when you would stop. Those three answers become your green, yellow, and red.
Use it today
- Email suppliers about what's missing from their quotes: a script drafts the asks, and a person clicks Send.
- Related idea: Start with draft, earn trust.
From my training: this is how we set up review on every workflow we build. People watch the exceptions, not the whole pile.