Robot Brains — Local AI Compute, Measured
· Dgx Spark
/ Gb10 Inference Tuning
Measured throughput and memory for qwen2.5:7b and qwen3:30b-a3b at 8 to 64 concurrent requests on a GB10 Grace-Blackwell with 122 GiB unified memory. Includes three findings we could not find written anywhere else: the systemd cgroup does not account for unified memory, so a MemoryMax-based watchdog never fires; you must raise the inference server's parallelism before the client's or you are timing your own queue; and a thinking model under format:json can return an empty response until you pass think:false.
Robot Brains — Local AI Compute, Measured
· Agent Fleet
/ Local Agent Fleet Benchmarks
Operational numbers from 1,372 local LLM agent invocations across 47 research runs in one day on a DGX Spark: wall-clock per run, pages read, evidence yield and answer quality by subject. Includes the figure nobody publishes -- 61% of page-reading agents produced no usable evidence at all -- and the measured threshold at which a cheap text filter removes a quarter of those calls with zero false negatives. Zero external inference cost.
Robot Brains — Local AI Compute, Measured
· Failure Modes
/ Local Ai Failure Modes
Failure notes from running a self-hosted LLM research fleet on an NVIDIA DGX Spark (GB10): the wrong diagnoses, the invalid measurements and the guards that exist because something broke. Includes the one that cost a power cycle -- unified memory does not OOM-kill, it stalls the entire machine with no kernel message and no killed process -- plus a cgroup watchdog that cannot see the allocation that kills you, a reasoning model that returns empty output under a JSON grammar, and a GPU sitting at 0% behind a full queue. Eight of the ten are silent degradation, not crashes. Also: a model that is the fastest and highest-quality reader measured and still unusable, because it does not batch -- aggregate throughput flat from 16 to 32 concurrent requests while per-stream speed held at 87 tok/s.
Robot Brains — Local AI Compute, Measured
· Verification
/ Verifiers Beat Models At Catching Models
One evening in which a 200-line deterministic verifier caught three real defects in a production newsroom pipeline -- including one in itself -- while three local research runs asked to reason about the same problem all returned MIXED and one reproduced the exact failure it was asked to prevent, emitting its own instruction template as an answer and labelling it certainty 10/10. Includes the repo-bundled audit run that searched the open web for 'regex patterns used in content moderation systems' instead of reading the source files it was given, and handed the whole job back. What did survive with numbers: a 149M-parameter cross-encoder reranker worth +20 points Hit@1 (62% -> 83%), page-level chunking at 0.648 RAG accuracy, structured decoding that constrains format but not content, and no measured evidence for length control at all -- matching a word budget that was in the prompt twice and exceeded twice.
Robot Brains — Local AI Compute, Measured
· Evaluation
/ Benchmarks That Cannot Tell Models Apart
A six-item evaluation set rated four models from 7B to 32B at 5/6 or 6/6 -- a one-item spread across a 4.5x range of model size -- and hid a real defect in a production pipeline. What replaced it: 45 items across 14 named failure categories, deliberately imbalanced because false-accept is the expensive error, scored on balanced accuracy after verifying that always-yes and always-no both land at 50%. On its first run it found the production synthesis model scoring 0/3 on hedged sources -- reading 'reportedly aim to close in June' as support for 'will close in June' -- a qualifier-stripping behaviour the old set could not see because it contained no item of that kind. Also: a unit-mismatch blind spot shared by every model tested, and why that one wants a deterministic check rather than a better model.
Robot Brains — Local AI Compute, Measured
· Rover Autonomy
/ Rover Autonomy Without A Language Model
A tracked predator-deterrence rover (brain: a Jetson Orin Nano Super) shipped a fully offline autonomous-mission mode -- relative turn/drive steps, dead reckoning, no model involved -- after finding and fixing one real default (obstacle avoidance was off by default everywhere) and disproving one suspected bug (a preemption race that a code summary implied but the actual function already handled correctly). Separately, a 1.4B-parameter looped model (Ouro) validated clean on a spare DGX Spark: adjustable reasoning-depth per query confirmed by reading its code, latency scaling linearly from 0.76s to 5.7s across 1 to 8 loops at flat ~2.9 GB memory, and a clean 2-for-2 win over the model already in production on two of that product's own real, documented bugs. The smallest candidate for running that intelligence directly on the rover's own computer (Parcae-140m, chosen for being the smallest in its family) never loaded at all -- not a hardware ceiling, a packaging one: its publisher requires a newer Python toolchain than the one already keeping this exact board's camera and detection software running. Confirmed zero disruption to the live detection service across the whole attempt.
Robot Brains — Local AI Compute, Measured
· Ragflow Eval
/ Ragflow Citation Integrity
Built RAGFlow from source for aarch64 and pointed it at 321 real documents from our own research-fleet database -- 471 answerable claims, 95 deliberately unanswerable ones from unrelated topics (Prince Andrew, Tesla's valuation). Retrieval found the right document first 82.6% of the time (MRR 0.864). On citation integrity: zero of 136 citation markers pointed to a document that does not exist, and on the 60 genuinely unanswerable questions it refused all 60, including 9 where retrieval handed it evidence-shaped chunks anyway. Includes a self-caught grading bug: an automated classifier scored 15% of answerable questions as refusals; reading the actual text found most were real answers hedged with 'however' -- corrected true refusal rate 3.3%. Also, independently: re-checking 659 of our own fleet's stored quotes against source pages as part of building the gold set found 74.8% byte-exact and 91.7% exact after normalization, a second confirmation of a property we'd only measured one other way before.
Robot Brains — Local AI Compute, Measured
· Reliability
/ Keeping The Spark Up
A full incident history for one DGX Spark (GB10, unified memory, aarch64, CUDA 13): 13 catalogued failures grouped by root theme -- unified-memory over-commitment, silent CPU fallback, a queue that reported honestly idle over unfed work (raised 8+ times), a measuring instrument that was wrong rather than the model (5 times), and environment/scheduling drift. While compiling it, the benchmark it's built from failed three more times in real time: a PYTHONUNBUFFERED gap that loses output on an external signal, and a GPU allocation failure precisely isolated to one condition -- loading a ~19GB custom model at the production 32,768-token context, reproduced three times including on a freshly restarted, zero-resident, 108 GiB-free daemon, while a parallel check on a similar model logged the identical kernel warning and still succeeded. Includes the exact line of Ollama's own source code responsible (a hardcoded 5-second eviction-confirmation timeout that proceeds regardless), a previously undocumented GB10 thermal hard-poweroff mechanism with a sourced fix, and six sourced, ranked changes -- none of them 'retry harder.'