We run a fleet of small AI models on a single NVIDIA DGX Spark — a desktop-sized machine with 122 gigabytes of memory shared between its processor and graphics chip. The models search the web, read pages and write reports with citations. Everything runs locally. Nothing is sent to any company's servers and no usage is billed.
To decide which models to actually use, we measured them. Here is what came back, in plain terms.
First, a name that gets confused constantly
We tested Nemotron, which is NVIDIA's family of openly available language models. We did not test NeMo, which is NVIDIA's software framework for training models. The names are similar, they come from the same company, and they are routinely swapped in coverage. The specific version here is nemotron-3.5-lightning:30b-a3b-q4_K_M — roughly 30 billion parameters, compressed so it fits comfortably in memory.
Test one: can it catch a lie?
This is the test that matters for our work. The fleet's job is to read a web page and pull out facts, and the dangerous failure is not missing something — it is confidently repeating something the page never said. So we built a set of 56 short items, each pairing a claim with a piece of evidence. Some of the claims genuinely follow from their evidence. Most do not, in sixteen different ways: a swapped name, a changed number, a changed unit, a reversed negative, a leap from correlation to cause, or a quote that is accurate but irrelevant.
The set is deliberately weighted toward claims that should be rejected, because letting a false claim through is the expensive error — that is how an invented fact ends up in a published report with a citation attached.
That weighting creates a trap, and defeating it is the whole point. A model that simply answers "no, not supported" to every single item — understanding nothing, reading nothing — scores 76% on plain accuracy. It looks like a solid B. So plain accuracy is useless here. The headline number below is balanced accuracy, which averages how well a model spots real support and how well it spots the absence of it. Answer "no" to everything and you score 50%. Answer "yes" to everything and you also score 50%. You only climb by actually distinguishing the two.
The results
| Model | Balanced | Plain | False accepts | Seconds per check |
|---|---|---|---|---|
| Qwen3.8-27B (uncensored build) | 93% | 89% | 6 of 43 | 4.2 |
| NVIDIA Nemotron 3.5 Lightning | 92% | 88% | 7 of 43 | 0.7 |
| Qwen3.8 (abliterated build) | 92% | 88% | 7 of 43 | 1.9 |
| Qwen3.6-27B | 91% | 86% | 8 of 43 | 3.6 |
| Mistral Small 3.2 | 91% | 86% | 8 of 43 | 2.1 |
| Qwen2.5 7B | 89% | 88% | 6 of 43 | 0.7 |
| Qwen2.5 14B | 87% | 93% | 1 of 43 | 1.5 |
| GLM 4.7 Flash | 85% | 77% | 13 of 43 | 0.6 |
| Qwen3 30B-A3B | 85% | 77% | 13 of 43 | 0.4 |
| Our own tuned model | 78% | 75% | 12 of 43 | 2.5 |
| Patronus Lynx 8B | 68% | 75% | 8 of 43 | 2.0 |
Four things worth pulling out
Nemotron's real result is the speed, not the score. It tied for second on accuracy at 92%, one point behind the leader. But it reached that in 0.7 seconds per check against the leader's 4.2 — six times faster for one point of accuracy. When a single research run makes hundreds of these checks, that ratio decides what is practical to run at all. It was also the only model in the top group that was simultaneously among the fastest.
Size did not predict accuracy. A 7-billion-parameter model scored 89%, beating a 30-billion-parameter model at 85%. Nemotron was strongest of all eleven at spotting opinion presented as fact, getting every such item right. Its weak spots were narrower: evidence that contradicts itself, changed units of measurement, and claims where a date has quietly gone stale.
The most accurate-looking model was actually the most timid. Qwen2.5 14B posted the best plain accuracy in the table, 93%, and let through only one false claim out of 43 — by far the best on the error we care most about. Yet its balanced score is only 87%, below models with worse headline numbers. The reason is that it achieves its caution by rejecting things, including things that were properly supported. A verifier that refuses everything is safe and useless. This single row is the clearest argument for why we stopped reporting plain accuracy.
The purpose-built tool finished last, with important context. Patronus Lynx is built specifically to detect AI fabrication, and it scored 68% — bottom of the table. It also failed to produce a usable research plan and dropped the numbered citations out of its output, which for our pipeline is disqualifying on its own. In fairness, it is an 8-billion-parameter model measured against 27- and 30-billion-parameter models, and it was designed for a narrower task than the general agent work this test covers, so this is not a like-for-like comparison of its intended job. We also include our own tuned model at 78%, below every stock model we tested, because leaving that out while reporting someone else's poor score would not be honest.
Test two: raw speed
Separately, on 2 September, we measured how much text Nemotron can produce when many jobs run at once. A "token" is roughly three-quarters of a word.
| Jobs at once | Total output | Per job | Reading speed | Memory used |
|---|---|---|---|---|
| 8 | 64.7 tokens/sec | 86.9 | 1,163 tokens/sec | 38.7 GB |
| 16 | 84.9 tokens/sec | 87.8 | 1,272 tokens/sec | 41.1 GB |
Doubling the workload from eight simultaneous jobs to sixteen raised total output by about 31% while each individual job got marginally faster rather than slower, and cost only 2.4 gigabytes of additional memory. That is the useful finding: on this hardware the machine was not saturated at eight jobs, and the extra parallelism is close to free. Graphics memory use did not move at all between the two runs.
What this does and does not tell you
These are our numbers, on our hardware, for our task. The accuracy test measures one specific skill — deciding whether a short claim follows from a short piece of evidence — which is central to what we do and is not the same as general intelligence, writing quality, or coding ability. A model that wins here might lose badly at something else.
The measurements were also taken at a fixed context length of 24,576 tokens, and the speed figures do not carry over to other settings, because the memory a model needs scales with how much text you ask it to hold at once. Anyone repeating this should state their context length, or the throughput numbers mean nothing.
We publish the ones that make us look bad alongside the rest. Our own tuned model came tenth of eleven.
