Your benchmark probably cannot tell a 7B from a 32B. Ours could not.
Document text
Your benchmark probably cannot tell a 7B from a 32B. Ours could not.
How a six-item evaluation set hid a real defect in a production AI pipeline for weeks, what replaced it, and the defect it found on its first run. Measured on a self-hosted research-agent fleet, September 2026.
There is a particular kind of wasted work in evaluating language models: you build a test, every model passes it, and you conclude the models are equivalent. They are not. Your test is.
We ran a six-item set to score the reasoning stages of a local research pipeline — planning, claim verification, synthesis. Four models, from 7B to 32B parameters, scored 5/6 or 6/6. A one-item spread across a 4.5x range of model size. We read that as "size does not matter for these stages," which is a conclusion about models, drawn from what was actually a fact about the instrument.
Then one model swept it 6/6 and we nearly promoted it on that basis. Running the same set against the models already in service showed the smallest model on the machine also scored 6/6, faster. The entire measured difference was one question out of six — a coin flip.
An unbeaten score on a small benchmark is a statement about the benchmark.
What was wrong with the set
Six items is the obvious problem, and the least interesting one. Three others mattered more:
It tested one failure mode. Every negative item was a variant of the same thing — a claim that outruns its quote. Nothing tested whether a model notices a unit changing, a date, a negation, or a hedge.
It was balanced 3 yes / 3 no. A model that answers "supported" to everything scores 50% for free, and 50% looks like a model that is trying.
It reported one number. A single accuracy figure cannot tell you what a model gets wrong, and what it gets wrong is the only actionable part.
What replaced it
45 items across 14 failure categories, each category naming a specific way evidence can fail to support a claim:
| category | what it tests |
|---|---|
direct, paraphrase, multi_evidence |
genuinely supported claims, including reworded and multi-source |
entity_swap |
the quote is about a different product, company or jurisdiction |
number_mismatch |
the figure does not match |
unit_mismatch |
the figure matches, the unit does not — 240 bar vs 240 psi |
causal_leap |
the quote states a fact; the claim asserts a cause |
overgeneralize |
one config line or one trial read as a general rule |
negation |
the quote says the opposite |
temporal |
true once, claimed as current |
hedged_source |
the source hedges — "reportedly", "may", "up to" — the claim asserts |
partial |
the quote covers half a compound claim |
contradictory |
two evidence lines disagree with each other |
irrelevant |
the quote is true and says nothing about the claim |
Deliberately imbalanced: 11 supported, 34 not. In a research pipeline the expensive error is false-accept — an unsupported claim entering a cited report. Rejecting a good claim costs one fact. Accepting a bad one costs the document's credibility.
That imbalance breaks raw accuracy, so the headline metric is balanced accuracy — the mean of the two per-class recalls. We verified this rather than assuming it: always-yes, always-no and always-partial all score exactly 50%, while always-no scores 76% raw. If your imbalanced benchmark reports raw accuracy, a model that refuses everything is beating your models.
Scores are reported per category, and each model's weakest categories are named.
What it found on the first run
| model | balanced | raw | false-accept | weakest categories |
|---|---|---|---|---|
| 7B | 91% | 91% | 3/34 | unit_mismatch, multi_evidence |
| 14B | 86% | 93% | 0/34 | multi_evidence, direct, paraphrase |
| 30B MoE (in production) | 87% | 80% | 9/34 | hedged_source, partial, unit_mismatch |
| 32B MoE | 93% | 89% | 5/34 | contradictory, unit_mismatch, temporal |
A 7-point spread in balanced accuracy and a 0-to-9 spread in false-accepts, where the old set had produced a one-item spread. The instrument now discriminates.
The defect
The 30B MoE was the production model — running planning, reduction and synthesis. It writes
the prose a human reads. It scored 0/3 on hedged_source. Given:
claim: The merger will close in June. evidence: "The companies reportedly aim to close the merger sometime in June."
it answers supported. All three items of that kind, wrong. It also took 1/3 on partial.
Those are one behaviour: it strips qualifiers. "Reportedly aim to" becomes "will". "Up to 30 percent" becomes "30 percent". "Covers dental" becomes "covers dental and vision". That is precisely the failure a citation-anchored pipeline exists to prevent, happening in the stage that produces the finished text — and its overall raw accuracy, 80%, reads as unremarkable rather than alarming.
The old benchmark rated this model 5/6, indistinguishable from every other model, because it contained no hedged-source item at all. The defect was not introduced. It was always there, and the test could not see it.
A blind spot every model shared
unit_mismatch was the weakest category almost everywhere. Given "240 psi" claimed against a
quote reading "240 bar", most models match the number and ignore the unit — a factor of 14.5.
Only one model of four scored 3/3.
This is worth separating from the rest, because it suggests the fix is not a better model. A unit comparison is deterministic; a model call is not. The same reasoning previously replaced a model-based prompt-injection detector with a regex, after the same page was flagged on one run and cleared on the next. Where a check can be deterministic, a model is the wrong tool — not because it is worse on average, but because it is unreliable in a way you cannot bound.
If you take three things
- Run your benchmark against the models you already trust before you use it to justify a change. A baseline you have not run is not a baseline, and a score with nothing to compare against is how a one-item spread becomes a promotion decision.
- Count the failure modes your set contains, not just the items. Ours had 45 items but the number that mattered was 14 categories. The defect lived in a category the old set did not have — and no amount of adding more items of the same kind would have found it.
- If your set is imbalanced, report balanced accuracy, and check what a degenerate strategy scores. Ten minutes of work. It tells you the floor your models actually have to beat.
Every number here was measured on the hardware and pipeline described; none was generated by a model. The evaluation set and the harness that scores it are ordinary Python and could be rebuilt from this description in an afternoon — the transferable part is the categories and the imbalance, not the code.