Nothing on this box has ever failed loudly. Thirteen incidents, and a live fourth one caught while writing this sentence.

A full incident history for one NVIDIA DGX Spark (GB10 Grace-Blackwell, unified memory, aarch64, CUDA 13), the fixes we found sourced from Ollama's own code and NVIDIA's own forums, and the exact GPU allocation failure that reproduced three more times while this was being written. September 2026.

A cold-boot race silently dropped inference to CPU for nine hours while the health check answered normally the whole time. A report-writer bug destroyed 11 of 12 models' results while the script reported success on every run. A GPU allocation failure killed a 13-model benchmark batch after five — systemd logged it as "Deactivated successfully." Thirteen incidents, grouped by theme rather than just date, because three of them took multiple layers of fixing before the real cause surfaced.

This has a name in the reliability literature: gray failure — "at least one app observes the system is unhealthy, but the observer sees it as healthy" (Huang et al., Microsoft Research, HotOS 2017). Every fix below either makes a failure impossible, or makes it loud. Nothing here just retries harder.

The catalog

A. Unified-memory over-commitment

THE WEDGE (2026-09-03, closed). A benchmark tuned at 4K context shipped 48 concurrent slots to production, which runs 12K–32K context. KV cache scales with context: 48 slots × 32K ≈ 130 GiB demanded on a 122 GiB box. The machine froze solid — ping answered, SSH did not — no OOM-kill, no log. Cost a power cycle. Fixed: a budget check now refuses an over-committed slot count before anything loads.

Reload-thrash froze the box twice in one week (2026-09-05, closed). One model was asked to run at three different context sizes depending on task, forcing ~160 full reload cycles in two days — each one grabbing several GB in a single instant. Tripped the memory safety shutoff repeatedly, exceeded it twice. Fixed: unified to one context size for every task that needs that model.

Silent mid-benchmark death via raw GPU allocation failure (2026-09-07, open — see the live section below). A kernel-level NV_ERR_NO_MEMORY fired loading model 6 of 13. The bash script died instantly — no traceback anywhere, systemd recorded a clean exit. 8 of 13 planned runs silently never happened.

B. Compute silently falls back to CPU

Nine hours of inference at 1/100th speed, invisible to every check (2026-09-03, closed). A cold-boot race let the inference daemon probe for GPUs before the device nodes existed, fell back to CPU, and stayed there — while its own health endpoint answered normally and the daily digest still generated reports. Root cause: a device-permission list that omitted the specific nodes CUDA needs, closing the door to everything not explicitly named. Fixed: startup now waits for the real device node before proceeding and fails loudly if it never appears; a runtime guard separately refuses to proceed when a resident model reports zero GPU memory.

C. "No work available" — reported honestly, over a queue nobody finished feeding

Raised as a complaint eight or more separate times. Each fix was real; each one turned out to be one layer of a deeper stack.

The dispatch timer built for exactly this was simply disabled (2026-09-04, closed). 20+ topics genuinely queued, GPU at 0%, and the timer meant to spend spare capacity on benchmarks instead of sleeping had never been turned on. Enabled it; it dispatched within seconds.

One layer deeper: the bench queue itself was hand-maintained and stale (2026-09-04, closed). The dispatcher decided what needed benchmarking from a tracking file that had to be manually updated — it knew about 2 of 12 models actually in the store. Fixed: the dispatcher now diffs the real store against the tracker on every tick and appends anything missing, automatically, permanently.

Named as a standing risk, not closed as a category (ongoing, partial). When this box looks idle again, check whether a new layer of this same defect exists before assuming the last fix regressed.

D. The measuring instrument was wrong, not the model

A 6-item test couldn't tell a 7B from a 32B (2026-09-03, closed). Every model scored 5/6 or 6/6 — a one-item spread across 4.5x the parameter count, which is noise dressed as a ranking. Replaced with a 45-item set across 14 failure categories (since grown to 56 across 16), deliberately imbalanced because a false-accept is the expensive error in a cited pipeline.

A text-normalization bug manufactured false fabrication findings (2026-09-03, closed). Whitespace was collapsed before punctuation substitution, so a real quote that had merely been reflowed scored as "absent" — which the production guard reads as fabrication and silently drops. Fixed at the root; a regression test now also asserts a genuinely invented sentence still scores absent, so the fix couldn't over-correct into accepting anything.

A report-writer bug destroyed 11 of 12 models' results in one run (2026-09-06, closed). The results file used a plain overwrite; one model per invocation meant each new model's write clobbered the shared file instead of adding to it. Only the last model's row survived. Fixed same day: reads existing same-day results, merges by model, then writes.

E. Environment drift and scheduling gaps

One config number lived in five different files (2026-09-04, closed). The daily job-limit was copied into five files, three carrying a comment to keep them in sync. One was changed; four weren't. The fleet refused jobs at a number nobody chose, and an automated status email reported "stuck" when the real state was an empty queue. Fixed: it now lives in exactly one place, queried live by every consumer.

A scheduled job failed in 12 milliseconds on a missing environment variable (2026-09-07, closed). A one-time job scheduled via a transient systemd-run unit died instantly — that mechanism doesn't inherit a normal login environment, and the script correctly refused to guess at an unset $HOME rather than silently doing something wrong. The scheduling itself looked completely fine: the timer showed armed, then fired, on time. Fixed for this job by passing the environment explicitly; the general fix is to stop using one-shot scheduling for anything that needs to look like a real session.

Two model-benchmark processes killed by an external SIGTERM, cause uncaptured (2026-09-07, open). Not an OOM kill — memory was healthy immediately after, and the serving daemon's own uptime was unaffected. Something sent a termination signal to two separate model loads in one run, and nothing recorded what.

Live: the report caught its own subject mid-incident, three separate times

Compiling this catalog and re-running the benchmark it's built from produced three more failures of the exact pattern this piece is about, in real time:

• A PYTHONUNBUFFERED-related output loss. research's CLI has a plain #!/usr/bin/env python3 shebang; run manually (not through the systemd unit, which sets PYTHONUNBUFFERED=1), Python's stdout is fully block-buffered against a non-TTY pipe. A process killed by an external signal — SIGTERM does not flush pending output — loses everything already printed with no way to recover it after the fact. This is a distinct, previously-uncatalogued mechanism from "the script died with no traceback": here, the script may have been about to print useful diagnostics, and the signal deleted them before they left the buffer.

• A precisely reproduced GPU allocation failure, isolated to one specific condition. Loading newsroom-bob:v1 (a ~19 GB custom Modelfile build) through the fleet's own benchmark path — which requests the production planning-stage context length, 32,768 tokens — failed with NV_ERR_NO_MEMORY three separate times, including on a freshly restarted daemon confirmed to be holding zero resident models with 108 GiB free. The same model loaded and answered correctly, twice, when called directly against Ollama's API at a small default context. The safety check that is supposed to catch this (llm.preflight()) estimates memory from model file size times a blanket 1.6 multiplier — it does not look at the actual context length a specific call is about to request. For most models tested tonight that multiplier held. For this one, at this context, it did not, three times in a row.

This is not a fully closed diagnosis. A parallel check on qwen32-voice:v1 (also ~20 GB, also called at 32,768 context) logged the identical kernel-level NV_ERR_NO_MEMORY warning at the same moment — and that request still completed successfully. So the failure is real and reproducible for one model, correlated with large-model-at-full-context loads generally, but not yet shown to be deterministic across every model that shape describes. Reported at this resolution rather than rounded up to a tidier, falser conclusion.

The fixes, sourced

Stop trusting eviction. Ollama's own confirmation gives up after 5 seconds. Reading Ollama's scheduler source directly: before unloading a model it polls for free memory to recover for up to 5 * time.Second — a hardcoded constant, not configurable — then proceeds anyway if memory hasn't converged, logging only a warning. /api/ps going empty confirms Ollama's internal bookkeeping dropped the model; it does not confirm CUDA has actually reclaimed the memory. Source: server/sched.go, ollama/ollama, main branch.

When a model is truly wedged, no client-side wait fixes it. Restart the server. Three separate GitHub issues on Ollama's own tracker describe a model stuck in a Stopping... state indefinitely, confirmed by an Ollama maintainer as the runner process getting wedged — and in every reported case, the only thing that reliably cleared it was killing the runner or restarting the service. Sources: ollama/ollama#7606, #9926, #13235.

On this specific hardware, no memory number can be trusted as a hard gate. An NVIDIA staff account confirms nvidia-smi memory reporting is N/A by design on this hardware — unified memory has no dedicated pool for the tool to report on, and the recommended workaround is ordinary system tools (top/htop/free) instead. A second, separate forum thread tracing the "unresponsive zombie instead of CUDA OOM" failure shape finds a different, kernel-level gap: the OS doesn't recognize GPU-side unified-memory allocations as memory pressure at all — a community technical analysis in that thread, not an official NVIDIA statement, describes it as "the kernel doesn't know that ~71GB of 'used' memory is GPU allocations, so it doesn't feel memory pressure." A third, independent forum trace found standard cgroup accounting also misses GPU-side unified allocations — a measured 10.7 GB GPU allocation moved cgroup's own counter by 377 MB. Three different reporting paths, three different specific gaps, the same conclusion: every memory number on this hardware is a soft, advisory signal, not something safe to gate a script on directly. Sources: NVIDIA forums — nvidia-smi on DGX Spark, DGX Spark unresponsive-instead-of-OOM thread.

The freeze may not always be memory. This box has a documented thermal kill-switch. Independently reproduced by a third party: the GB10's embedded controller performs a hard power cut on thermal spike faster than Linux can log anything — a hotspot can hit 98°C in about two seconds, with no software throttle engaging below the critical point. A second, independent NVIDIA forum thread reports the identical symptom under sustained concurrent-model load. The documented fix: sudo nvidia-smi -lgc 300,2200 caps boost clock and gives thermal management room to engage first, at roughly a 5% throughput cost, persisted via a systemd unit since it doesn't survive reboot alone. Source: DGX-Spark-Hard-Poweroff-Fix.

Make every death produce a log line — including SIGKILL's one blind spot. set -euo pipefail does not catch a backgrounded process killed by an external signal — that's outside what errexit ever watches. The actual fix is a trap that fires on any exit, and a signal-specific trap that logs before dying. SIGKILL itself cannot be trapped, by Unix design — no bash-level fix exists for that specific signal; a backgrounded child killed by a signal returns 128 + signal number from wait, which is already available and mostly unused. Source: BashFAQ/105.

A completion count that can't be faked by a short file. The report-overwrite bug and the dropped-8-of-13 run share one shape: the run's own summary can't be trusted, because whatever silently failed also silently skipped writing its own honest accounting. The fix is a manifest checked from outside the run: know the expected count going in, append one line per completed item as it finishes, and let a completely separate process compare "lines logged" against "items planned." Pattern: healthchecks.io — monitoring cron jobs.

Run anything whose output matters, and might be killed, unbuffered. python3 -u, or PYTHONUNBUFFERED=1, whenever a script's diagnostic value depends on output surviving an abnormal death — confirmed necessary tonight, the hard way, three times.

What changes, in order

1. Add an explicit settle delay (≥8 seconds — Ollama's own internal ceiling is 5) after every model unload, instead of trusting /api/ps going empty.
2. Restart the Ollama service every N models in a sweep, not just request unload — the only fix that reliably clears a wedged runner across three concordant upstream issues.
3. Make llm.preflight()'s memory estimate context-aware per call, not a blanket multiplier — the gap this session's live incident actually found.
4. Add signal traps and wait-exit-status decoding to every long-running script, and run anything whose failure output matters unbuffered.
5. Cap GPU boost clock and persist it via systemd — closes a plausible second cause of past freezes that memory fixes alone can't touch.
6. Per-item heartbeat file plus an independent staleness/count checker — the one mechanism that would have caught every incident above from outside the failing process itself.

Incident catalog compiled from this fleet's own operating history, plus a fourth live incident found and precisely diagnosed while writing this piece, in real time, on the actual machine it describes. No model wrote this; the failures were ours.