Twelve ways a local AI agent fleet breaks, and how each one was found

Failure notes from building and running a self-hosted LLM research fleet on one NVIDIA DGX Spark (GB10 Grace-Blackwell, 122 GiB unified memory, aarch64, CUDA 13), August–September 2026. Every entry below actually happened on this machine. Several cost hours; one cost a power cycle.

Most write-ups about local AI describe the version that worked. That is the least useful half. What follows is the other half: the wrong diagnoses, the measurements that were invalid, and the guards that now exist because something broke.

Each entry gives the symptom you would actually see, the wrong answer we reached first where we reached one, the real cause, and the guard — the check that makes the failure impossible or loud rather than silent.

Companion documents: Tuning an NVIDIA DGX Spark (GB10) to serve many concurrent local LLM agents and What a local research-agent fleet actually delivers.

1. Unified memory does not OOM-kill. It hangs the whole machine.

Symptom. The box stopped. ICMP still answered. The inference server's /api/tags returned 200. /api/generate never returned. SSH opened a TCP connection and then never sent a banner. No OOM killer message, no kernel log line, nothing in the service journal. The machine was not dead and was not alive.

Wrong answer. "This is not an out-of-memory freeze" — because every OOM either of us had seen leaves a corpse: a killed PID, a Killed process line in dmesg. There were none, so we looked at the network and the service first, and were wrong for about twenty minutes.

Real cause. On a Grace-Blackwell unified-memory system the GPU allocates from the same pool as the OS. Over-commit does not trigger the OOM killer the way an ordinary process would; it drives the machine into an allocation stall where nothing can make forward progress, including sshd accepting a login. The only exit was a power cycle.

What made it happen. KV cache scales linearly with context length. We had benchmarked at num_ctx 4096 and then shipped the slot count that benchmark justified while production ran at 12288–32768. Same slots, six times the cache per slot.

Guard. Refuse to start rather than discover this at runtime:

def budget_check(per_slot_gb_at_24k: float = 1.3, reserve_gb: float = 25.0) -> str:
    worst_ctx = max(CTX[k] for k in ("read", "plan", "reduce", "synth"))
    kv   = SLOTS * per_slot_gb_at_24k * (worst_ctx / 24576)
    need = kv + 45.0                       # + weights
    if need > total_gb - reserve_gb:
        raise SystemExit("REFUSING TO START: " + verdict)

The transferable rule: benchmark at the context length you will actually run. A throughput number measured at a toy context is not a smaller version of the real number — it is a different number, and using it to size slots is how you wedge the machine.

2. A MemoryMax= watchdog cannot see the allocation that kills you

Symptom. A systemd MemoryMax=6G limit and a cgroup-based watchdog were in place. Neither fired during the stall in §1.

Real cause. MemoryCurrent accounts for the cgroup's own charged pages. Unified-memory allocations made by the GPU driver on behalf of the model are not charged there. The watchdog was reading a number that stayed small while the machine ran out of memory.

Guard. Watch MemAvailable in /proc/meminfo — a system-wide figure — and act on a stall line above zero, not on the cgroup's view. On this class of hardware the cgroup is the wrong instrument, and it fails quietly, which is worse than failing loudly.

3. Measuring client concurrency against a server pinned lower

Symptom. A concurrency sweep that looked clean, published, and was wrong within a day. Going from 8 to 48 concurrent clients showed a suspiciously modest gain.

Real cause. The inference server was pinned at 8 parallel slots. Above 8, we were timing our own request queue, not the GPU. The published claim ("slots are nearly free") was falsified the same day.

Guard. Raise the server's parallelism first and confirm it in the service environment before varying the client's. When the two disagree, every number above the server's limit is a measurement of your own queue. The superseded results now carry a correction banner on top rather than being deleted — a wrong number that someone already read should be corrected in place, not disappeared.

4. GPU at 0% with a full queue — starved by the non-GPU stage

Symptom. Hundreds of tasks pending, sixteen LLM slots free, GPU utilisation at 0%.

Real cause. Two compounding errors. Six HTTP fetch workers were feeding sixteen LLM slots, so the readers had nothing to read. And the fetch timeout was per-hop, not total: a chain of slow redirects held one worker for 61–72 seconds while each individual hop stayed under its limit.

Guard. A total deadline per fetch, not per hop; fetch workers raised to 48 against 16 LLM slots. GPU went from 0% to 95%.

The transferable rule: in an agent fleet, the bottleneck is almost never the model. It is the I/O stage in front of it. Size the cheap stage several times larger than the expensive one, and instrument per-stage rates — an aggregate "tasks per minute" number hides this completely.

5. A thinking model returns empty output under a JSON grammar

Symptom. A 30B MoE scored 0.0% on every extraction metric — not badly, but zero, with no errors reported. Same prompts, same pages where a 7B model worked fine.

Real cause. It is a reasoning model. Under format: json it spent its token budget in the thinking channel and returned empty content. The API call succeeded. The response was blank.

Guard. Ask the server what the model is, and suppress thinking only for models that have it:

caps = show(model).get("capabilities", [])
if "thinking" in caps:
    payload["think"] = False

This recurred with a second, unrelated model from a different vendor months later — the same empty-content signature — which is why it is worth naming as a class of failure rather than a quirk of one model. If a model scores exactly zero with no errors, check the thinking channel before you conclude anything about its ability.

6. A quote verifier that a short quote walks straight through

Symptom. Nothing visible. Found by a self-test, not in production.

Real cause. Extracted facts were required to carry a verbatim quote, verified by substring match against the source. Short quotes — three or four common words — match almost any page by accident, so a fabricated fact carrying a short quote passed verification.

Guard. Quotes under five words get an exact check rather than a fuzzy one. Fabricated-quote rate across the fleet went from 5.5% to 0%.

7. A prompt-injection detector that is itself a model call

Symptom. The same page was flagged as an injection attempt on one run and not the next.

Real cause. The detector was a model call. A non-deterministic guard against a deterministic attack means an attacker gets as many attempts as they like, and your incident log is noise.

Guard. A regex over known injection shapes, applied before the content ever reaches a model, with quarantine on match. Deterministic in, deterministic out. The model that reads fetched pages has no tools, no network and a strict output schema — the containment is structural, not behavioural, because a model instructed not to obey injected text is still a model being asked to make a judgement call about text designed to fool it.

8. Nearly publishing the setting that hung the machine

Symptom. A draft tuning guide recommended, in good faith, the exact configuration from §1.

Real cause. The measurement was real and reproducible. It was taken at a context length no production workload used. Reproducible and correct are different things.

Guard. That guide now leads with the cautionary table instead of the recommendation. Worth stating plainly: the most dangerous document is an accurate benchmark presented without its conditions. Anyone applying it inherits your assumptions without knowing they exist.

9. Restarting the inference daemon during live work

Symptom. A batch of runs failed at the planning stage, all at once, for no apparent reason.

Real cause. We stopped the inference service to pull a new model while runs were active.

Guard. Model pulls wait for an idle fleet; the queue heals with a retry path rather than abandoning the run. Separately, a permanently-failing task used to requeue forever — that now has an abandonment path, because an infinite retry loop is a failure that looks like activity.

10. Three small ones that each cost real time

• timeout does not exist on macOS. A verification script reported failures that were the verifier failing, not the thing under test. When a check fails, confirm the check runs.
• Working-directory drift in a long session wrote a results file into the wrong directory. Use absolute paths in anything long-running.
• Remote names recalled from memory instead of read were wrong twice. Read the config.

11. A model that is fast per stream and cannot batch at all

Symptom. A newly released 30B-class mixture-of-experts model — NVIDIA's nemotron-3.5-lightning:30b-a3b (31.6B total, ~3.6B active, nemotron_h_moe), served through Ollama 0.33.2 — benchmarked at 98 tok/s single-stream — faster than the vendor's own published range for this hardware, and faster than anything else on the box. On extraction quality it was the best model we had measured: 2.3x the evidence yield of the incumbent reader at a third of the latency, with a third of its fabrication rate.

Wrong answer. On those two numbers it was an obvious upgrade for the page-reading stage.

Real cause. Throughput did not scale with concurrency at all:

Doubling concurrency from 16 to 32 moved total throughput by −0.6%, while per-stream speed held at 87 and GPU memory never moved off 28.6 GB — 38% of the machine. That is not a saturated GPU. It is a server handling requests essentially one at a time: this hybrid Mamba/attention MoE architecture was not batching on this inference build. To be fair to the model, this is very likely an Ollama/llama.cpp limitation for nemotron_h_moe rather than a property of the weights — recurrent state in the Mamba layers is the usual reason a runtime falls back to serial. A different serving stack (vLLM, TensorRT-LLM) may well batch it fine. The lesson is about how you measure, not about whose model it is. Against the models actually in service it was 5.7x to 7.8x slower in aggregate, on the one stage that is entirely parallel.

Guard. Never accept a single-stream benchmark as a throughput result. Sweep concurrency and check that aggregate rises; if aggregate stays flat while per-stream stays high, the server is serialising and the model cannot serve a fleet no matter how good it is.

The transferable rule: per-stream speed is not throughput. For anything running many agents at once, the only number that matters is aggregate tokens per second at your real concurrency. A model can be simultaneously the fastest and the least usable thing you own.

Footnote to §11: the nvfp4 tag will not load on a Blackwell box

While pulling the model above, the quantisation that should be ideal for this hardware failed:

$ ollama pull nemotron-3.5-lightning:30b-a3b-nvfp4
Error: this model requires MLX support, but the MLX runtime is not available

MLX is Apple Silicon. Despite the nvfp4 name, and despite GB10 being Blackwell with FP4 support in hardware, that tag is an MLX build and cannot run on aarch64 CUDA. NVFP4 Nemotron is not reachable through Ollama on a DGX Spark at all — it would take vLLM or TensorRT-LLM, not a tag swap. q4_K_M is the working path. Worth knowing before planning around "NVFP4 on Blackwell": the tag name describes the quantisation, not the runtime that can execute it.

12. A benchmark too small to see the difference it is being used to justify

Symptom. The same model scored 6/6 on an agentic reasoning bench — a clean sweep, including two deliberately planted traps. We recorded that as evidence it should take over the planning and synthesis stages.

Real cause. Running the identical bench against the incumbents showed the smallest model on the machine — qwen2.5:7b — also scoring 6/6, and doing it faster at every stage than both the 32B and a 30B MoE. The entire spread across four models, from 7B to 32B, was 5/6 versus 6/6: one item, on a six-item set.

That is not a ranking. It is noise, and it had already been written down as a result because it was first seen in isolation, with nothing to compare it against.

Guard. A bench that cannot separate a 7B from a 32B is not measuring the thing you are using it to decide. Before any model swap is justified on a score: check the sample size, and check that the bench actually discriminates by running the models you already trust through it.

The transferable rule: an unbeaten score on a small benchmark is a statement about the benchmark. The moment a result is going to drive a change, the first question is not "which model won" but "can this instrument tell these models apart at all" — and a baseline you have not run is not a baseline.

The pattern underneath all twelve

Nine of these twelve are not crashes. They are silent degradation: a watchdog reading the wrong number, a benchmark measuring its own queue, a verifier passing fabricated text, a fleet at 0% GPU with a full queue, a model returning empty output with a success code. The system kept reporting that it was fine.

Local AI infrastructure fails quietly far more often than it fails loudly. The guards that matter are the ones that make a silent failure noisy — a start-up refusal, a per-stage rate you can actually see, an exact check instead of a fuzzy one — and the measurement discipline that makes you distrust a clean number until you know what conditions produced it.

Published so the next person debugging a hung Grace-Blackwell box, a zero-scoring reasoning model, or an idle GPU behind a full queue finds a written answer instead of an empty search result. Numbers came off the machine described. No model wrote this; the failures were ours.