Evaluation

You can't tell whether your AI setup got better

Here's a cheap way to test an AI change without a research team.

One task run six times split 4 to 2. My suite runs 89 cases five times each.

At the end of July, a user of a popular open-source AI project reported that its new version gave worse results. Others checked by running the same task once on each version. Then someone ran one task six times with identical inputs and got a 4-2 split, which means a one-off comparison can come out either way and proves very little. A lot of organisations adopting AI are judging their own changes on evidence that thin.

Teams usually can't tell whether a change to their AI setup made things better or worse. The models and tools change every few weeks and the same prompt can give a different answer on the next run, so the decision to keep a change tends to come down to how it felt. A model upgrade lands or someone rewrites the prompt library, and the review meeting runs on anecdotes. One person had a good week and credits the new setup, and another hit an odd edge case and stopped trusting it. Both are drawing conclusions from a handful of runs.

I've run an evaluation suite over my own agent workspace since June, and the method carries over to a team without much change. Three decisions do most of the work, and none of them depend on the tooling.

The first decision was to test only for things that had already gone wrong. Each of the 89 cases in the suite started as a real failure I wrote down when it happened, like an agent citing a file that didn't exist or quoting a number it had never looked up. Every case has a date and an incident behind it, which makes the suite much easier to defend than a list of guesses about what the AI might get wrong.

Second, nothing runs once. Each case normally runs five times, and the result is a pass rate plus a measure of how much it moves between runs. The catch is that a small improvement often won't show up, because it's smaller than the run-to-run noise. When that happens the honest answer is that I don't know whether the change helped, and I'd rather say so than keep it on a hunch.

The third was to keep every check mechanical. A check might confirm that the answer contains a required value, or that a cited file exists, but none of them ask another model for its opinion. Having one model grade another is common now, and it has its place for judging open-ended quality. For tracking regressions it causes a specific problem. The grader varies from run to run as well, so when the score drops you can't tell whether the workspace got worse or the grader did. Mechanical checks are cruder and miss plenty, but they behave the same way every run, so when a score looks wrong you can open the check and see exactly why.

None of this needs a research team. Start with the failures your people already hit, the ones that get pasted into Slack with "look what it did". Write up the best twenty as tests with answers a script can check. Run the set a few times against any proposed change, and only roll the change out if it beats the current baseline. The suite catches things that used to work and have stopped working, which is usually what you need to know before a rollout. Whether the output is actually good is still a judgement for people.

The measurement only matters if something acts on it, so I'd pair it with one governance rule. Every addition to your AI setup should come with a claim about what it improves and a date when someone will check. If the check can't show it helped, the addition comes out. Without a review date, AI configuration keeps growing, because adding something that sounds useful costs nothing and nobody is assigned to take things out.

I've published the full method, including the harness and the case format, in EVALUATION.md. And the next time someone tells you the new AI setup is better, ask them by how much.

Get new essays by email on Substack. This one is on Substack too.

← All writing