Evaluating AI agents

I recently built a suite of agents that did research for users. A user asks a research question, and the agents go and read what is out there and come back with an answer and its sources. I have written up how that got built separately, and this post picks up where that one leaves off.

A base agent splits the question, task agents work the directions in parallel, and the base agent writes the answer from their summaries.

The challenge was evaluating the agents.

No ground truth to check against

Earlier in my career I built classification models. Evaluating one of those is mechanical - you hold out a labelled set, you run the model, you count what it got right.

A research answer has no label. Two good answers to the same question can look nothing alike, so there is no reference text to compare against. And the answer on its own is not enough to judge - to know whether the references are any good you need to know what was out there to be found, which means reading the sources too. Reviewing a single answer honestly takes a long time.

Two options we dropped

The first was user ratings. We had thumbs up and down in the product, and it is tempting to treat them as a quality signal.

Two things go wrong when you evaluate on that. The bad tail is heavily over represented, so the ratings make the product look worse than it is. And almost nothing comes back from the middle, which is where most sessions live, so a regression that moves good answers to mediocre ones does not show up at all. That is the kind of regression a prompt change causes.

The second option was manual review, which is what we were doing. It is fine as a spot check. It did not scale to running on every prompt change, and prompt changes were happening several times a day.

The judge, and what it scored

So I proposed using LLM-as-judge. With help from the PM, we came up with a series of criteria that the LLM could judge on. Six of them, and they are not all graded the same way.

The first three are graded by counting and the last three by scoring. All the arithmetic happens in code.

We originally called the second one reference recall, which was the wrong name. Recall means "of the sources that existed, how many did you find", and answering that needs a reference set we never had. What we were actually measuring is how much of what the agent pulled the answer ended up using, so utilization.

For the first three the judge never produces a score at all, which is why they are the most reliable numbers in the suite. The rest of the design:

Call 1, the counting criteria

This one needs the whole trajectory, which is a large input - without the references the agent pulled, none of the three questions can be answered.

You are checking one research answer against the references the agent
pulled. Answer every question below yes or no. Do not score, do not rate,
and do not compute any ratios.

For each reference in the trajectory:
  relevant   does this reference bear on the question that was asked?
  used       does the answer draw on this reference?

For each substantive claim in the answer:
  grounded   is there a reference in the trajectory that supports it?

Return one object per question, with the quote that settles it:
{"id": "...", "check": "relevant|used|grounded", "answer": "yes|no",
 "evidence": "<quote>"}

The ratios get computed from those objects in code.

Call 2, the scoring criteria

This one needs only the question and the answer, so it is a small input.

You are scoring one research answer. For each criterion below, write the
reasoning first, then the score, then an evidence quote from the answer.
Do not produce an overall score.

Scores are 1, 2 or 3:
  1  fails the criterion
  2  partially meets it
  3  meets it

## effectiveness
Does the answer resolve the question that was asked, including the parts
that were implied but not stated? An answer that is correct but addresses
a neighbouring question scores 1.

## fluff
Tally the sentences carrying no information the user did not already have:
restatements of the question, hedges, filler conclusions. Then score.
Fluent is not the same as informative, and length is not evidence of
either.

## structure
Could the user find the part they came for without reading the whole
thing?

Return one object per criterion:
{"criterion": "...", "reasoning": "...", "evidence": "<quote>", "score": 1|2|3}

Validating the judge against the team

We turned the criteria into a judge prompt and validated it by having the team review a sample of the verdicts. Around 100 answers, each marked pass or fail by someone on the team, compared against what the judge said.

The overall agreement number looked acceptable. Splitting it by what the reviewer had said was less comfortable - we agreed on nearly all the answers the team passed, and on far fewer of the ones the team failed. The judge was lenient.

The reason was that the judge preferred longer answers, and it preferred fluent ones. An answer that was well written and comfortably off the point would get through. This is a known failure mode of LLM judges.

Two things in the prompt above came out of that. Asking for the reasoning before the score stops the model picking a number and justifying it afterwards. And the fluff criterion now asks for a tally of the padded sentences before the score rather than an impression, because whether an answer is padded was the question the judge was getting wrong.

The number we reported in v1 was raw agreement. Chance-corrected agreement prices that in, and the gap is large - the sample above is 82% raw agreement and a kappa of 0.52. Adding that was the next step and I did not get to it.

Running it on every change

I then set up an internal test suite using this eval that we ran for every prompt change. It ran in three layers.

Examples came out of production logs, with some synthetic ones for cases the logs did not cover.

The cost decides whether a suite keeps getting run or gets switched off. $5 on a commit is cheap enough that nobody argues about it. If we had put the 100 on every commit instead, someone would have turned it off inside a week.

What it caught

Mostly smaller regressions when we made prompt changes. A rewording of the base agent's planning instruction would move structure scores down, and because the scores were per criterion we could see which one moved rather than watching a single average wobble.

The repeated failure these tests surfaced was context blowing up during deep research, regularly pushing past 700k of our 1M window.

Looking at a trace, the cause was clear. Many tool results were large - fetched pages, PDFs - and a result never leaves once it is in the history. Three task agents reading their way through a question will put a few hundred thousand tokens of raw page text into context without anyone deciding that should happen.

This meant two things, and both showed up in the scores. The agent gets fewer back-and-forth loops before compaction. And the results get less reliable as the window fills.

Compaction was already in the system and we were reaching for it first, which is the wrong order - it is the only one of these options that loses anything. So I decided to write tool results to text files in Cloudflare R2 and include references instead, with a simple tool to fetch the full content when the agent needed it. This saved a lot of tokens without losing access to old knowledge, and it left compaction for the runs that still filled up. The elision rule and the stub format are in the build post, along with the thresholds we settled on.

The tradeoff

The tradeoff was an extra call to retrieve anything from history, which is latency the user pays for.

What we got was a much longer tool call history before compaction. I do not have before and after eval numbers on answer quality, so runway is the claim I can defend. Whether the extra runway made the answers better is something I would want to measure and did not.

If I did this again I would build the eval suite first. v1 was lenient and I knew it was lenient, but once it existed every prompt change had a number attached to it, and the context problem showed up as a pattern across runs rather than as a hunch. The next steps would have been chance-corrected agreement and a larger validation sample.