We made one small change to a comment moderation system. The pass rate jumped from 66.7% to 81.5% — a fifteen-point improvement. It also started approving abusive comments it had previously blocked. Both things were true at once, and only one of them showed up in the number.
What actually happened
We run an AI system that reads comments left on Facebook adverts and decides what to do with each one: approve it, hide it, or flag it for a person. Most comments are harmless. A meaningful minority are not.
The system was being a little too aggressive — hiding some perfectly ordinary comments, including sarcastic ones that were merely blunt rather than abusive. That is a real problem. Hiding a customer's genuine comment is its own kind of damage.
So we adjusted the instruction to make it more accommodating of tone. Then we re-ran the test set.
The pass rate went from 66.7% to 81.5%. By any headline measure, a clear improvement. If that number had been the only thing we looked at, the change would have shipped that afternoon.
It also let two abusive comments through — comments the previous version had correctly caught.
The change did exactly what we asked. It made the system more permissive. Being more permissive is correct for sarcasm and wrong for abuse, and a single accuracy figure cannot tell those two apart — because it adds them together.
Why an average hides the failures that matter
Most of the messages any business system handles are ordinary. Ordinary messages are also the easy ones. So when a change makes an AI more relaxed, it gets more of the easy majority right, and the average climbs.
The cases that would actually harm your business are, by definition, rare. They are a small share of the total, so their failures barely move the average. A system can get measurably better on paper while getting worse at precisely the thing you bought it for.
This is not unique to moderation. The same shape appears wherever an AI handles customer contact:
| System | The common case (moves the average) | The rare case (the one that hurts) |
|---|---|---|
| AI receptionist | "What time do you close?" | Someone describing symptoms that need a person immediately |
| Booking assistant | Straightforward new appointment | A cancellation that must actually free the slot in your diary |
| Enquiry chatbot | Price and opening-hours questions | A complaint that should never get a cheerful automated reply |
| Comment moderation | "Great, thanks!" | Abuse aimed at a customer under your own advert |
In every row, the left column is 90% of the volume and the right column is 100% of the risk. A vendor quoting you one accuracy percentage is averaging those together and handing you the result.
This is not a fringe concern. The NIST AI Risk Management Framework treats measurement as its own discipline precisely because deciding what to measure is where most of the risk actually sits — not in the measuring itself.
The thing that caught it
What caught this was not clever. It was a fixed set of example messages, each one labelled with what the correct outcome should be, kept in a file. Run the old version against it. Run the new version against it. Compare the two, case by case.
The comparison does not ask "is the new version good?" It asks a much more useful question: which cases used to be right and are now wrong?
In our case the comparison said, in effect: two things improved and two things broke, and the two that broke are the ones that get an advert reported. So we did not ship it. We went back and fixed the sarcasm problem a different way — one that did not buy its improvement out of the safety cases.
That is the entire discipline. It is not sophisticated. What makes it work is that it happens every time, before anything goes live, and that a bad comparison is allowed to stop a release.
Three questions to ask any AI vendor
If you are buying an AI system that talks to your customers, these three will tell you a great deal, quickly.
- "How do you know a change hasn't broken something that used to work?" There is a real answer to this and it involves a test set. "We test it thoroughly" is not it.
- "Can I see the results broken down by category, not one number?" If everything is reported as a single percentage, the failures are hidden by design — probably not deliberately, but hidden all the same.
- "What happens when it's not sure?" The right answer is that it stops and asks a person. A system with no route to a human will guess, and it will guess most confidently on exactly the edge cases you would want it to hesitate on. This is why our own AI receptionist hands over rather than improvising when it hits something outside what it knows.
None of these require you to be technical. They require the vendor to have thought about being wrong, which is a surprisingly good filter.
This is smaller than it sounds
"Evaluation" sounds like something with a budget attached. In practice, for a small business system, it is a few dozen real examples in a spreadsheet with a column for what should happen, and the habit of running them before every change.
Ours is not large and we do not pretend otherwise. It runs against a test suite rather than live traffic at scale, and the numbers in this article are from that suite. That is enough to have caught a change that would have gone out and embarrassed somebody — which is the whole point of it existing.
The uncomfortable version of this lesson is that the fifteen-point improvement was real. The change genuinely did make the system better at the thing we asked it to be better at. It just did so by trading away something we had not thought to measure. You do not find that by looking at the score. You find it by looking at what changed.
If you are weighing up an AI system for your business and want a straight answer about what it will and will not handle, that is a conversation worth having before you buy — not after.
Method note: the figures above come from our own comment-moderation system, measured against an internal test suite of labelled examples during development in August 2026. They are not production traffic statistics, and we would not present them as such.