We run a DGX Spark — an NVIDIA desktop box built around its GB10 chip and a large pool of memory shared between CPU and GPU rather than separate system and video RAM — to do local AI inference and research without sending anything to a cloud API. Over the course of about five days, it failed thirteen documented ways. A frozen machine that answered pings but not SSH. Nine hours of inference silently running on the CPU at roughly a hundredth of normal speed while every health check reported everything fine. A benchmark queue that looked completely empty when twenty topics were actually waiting behind it. A report-writer bug that quietly destroyed the results of eleven out of twelve models in a single run while the script reported success. While we were compiling the list of what had gone wrong, it produced four more failures in the same night, live.
Every one of these incidents shares a single property, and once you see it, you can't stop seeing it: nothing on this machine has ever failed loudly. That has a name in the systems-reliability literature — gray failure, defined by Microsoft Research as the condition where "at least one app observes the system is unhealthy, but the observer sees it as healthy." This is the catalog of thirteen instances of that pattern, and the specific, sourced fixes — pulled from Ollama's own source code and NVIDIA's own developer forums — that close the gaps still open.
The pattern in five shapes
Grouped by root cause rather than by date, because several of these took more than one round of fixing before the real cause actually surfaced.
A. Unified memory gets over-committed, and the machine simply stops responding. A benchmark tuned at a small context window shipped a slot count into production that runs a much larger one — memory usage for cached context scales with that window size, and the arithmetic quietly demanded more memory than the box has. The machine froze completely: pings answered, SSH did not, and there was no out-of-memory kill and no log entry anywhere. It cost a hard power cycle to recover. A second incident days later came from asking one model to reload at three different context sizes depending on the task, forcing roughly 160 full reload cycles in two days, each one grabbing several gigabytes in a single instant — that tripped an internal memory safety shutoff repeatedly and exceeded it twice. Both are now fixed: a budget check refuses an over-committed configuration before anything loads, and every task needing that model was unified onto one context size. A third, similar failure happened the same night we were writing this up — a raw GPU allocation failure while loading the sixth of thirteen models, mid-benchmark, killing the whole batch instantly with no traceback anywhere while the system's own service manager logged a clean exit. That one is still open, though we now understand the mechanism (see the fixes below).
B. Compute silently falls back to the CPU. A cold-boot race let the inference process check for available GPUs before the device files existed yet, fell back to running on the CPU instead, and then simply stayed there — for nine hours, undetected, because its own health-check endpoint kept answering normally and downstream reports kept generating on schedule. The root cause was a device-permission list that only allowed access to specific, explicitly-named hardware paths rather than the whole class of device — anything not on that list was invisible. It's fixed now: startup waits for the real device to exist and fails loudly if it never appears, and a separate runtime check refuses to continue if a model reports zero GPU memory in use.
C. "No work available," reported honestly, over a queue that was never actually finished being fed. This exact complaint was raised eight or more separate times, and each fix was real — each one simply turned out to be one layer of a deeper problem. First, a dispatch timer built specifically to fill idle GPU capacity with benchmark work had never actually been turned on; twenty-plus topics were genuinely queued and the processor sat at zero percent the whole time. Once enabled, it dispatched within seconds. The same day, one layer deeper: the thing deciding what needed benchmarking was reading from a tracking file that had to be updated by hand, and it only knew about two of twelve models actually present. The dispatcher now compares the real inventory against that tracker on every cycle and appends anything missing automatically. The pattern is explicitly flagged as a standing risk rather than a closed category — the lesson recorded internally is that the next time the queue looks empty, check for a new instance of this same defect before assuming an old fix broke.
D. The measuring instrument was wrong, not the model. A six-question benchmark couldn't distinguish a 7-billion-parameter model from a 32-billion-parameter one — every model scored either five or six out of six, a one-point spread across more than four times the parameter count, which is noise wearing the costume of a ranking. It was replaced with a much larger, deliberately imbalanced test across sixteen failure categories, weighted so that a confident wrong answer costs more than a correct refusal. Separately, a text-normalization bug in an internal fabrication-detection system collapsed whitespace before checking punctuation, so a real quote that had simply been reformatted scored as "absent" — which a downstream guard reads as fabrication and silently discards. Fixed at the root, with a regression test that also confirms a genuinely invented sentence still correctly scores as absent, so the fix couldn't accidentally swing the other way into accepting everything. And a report-writer used a plain file overwrite rather than an append, so running one model at a time meant each new model's result clobbered the previous one — only the last model in a thirteen-model run survived to the final file. Fixed the same day it was found.
E. Environment drift and scheduling gaps. One configuration number — a daily job limit — was copy-pasted into five separate files, three of them carrying a comment reminding whoever edited it to keep the copies in sync. One got changed; four didn't, and the system quietly refused work at a number nobody had actually chosen while a status report insisted things were "stuck" when the real state was simply an empty queue. It's fixed by putting that number in exactly one place, queried live by everything that needs it. The same night we were compiling this, a scheduled one-off job died in twelve milliseconds because the mechanism used to launch it doesn't inherit a normal login environment, and the script correctly refused to guess at a missing, required setting rather than doing something silently wrong with it — fixed for that job by passing the environment explicitly, with the durable fix being to stop using that scheduling mechanism for anything that needs to look like a real session. And in the same run, two separate model-load processes were killed by an external termination signal with the actual cause never captured anywhere — not a memory kill, since memory was healthy immediately afterward. That one remains genuinely open; what changed is that the script now logs and continues past a killed step instead of dying silently with it, which is a mitigation, not a diagnosis.
The fixes, checked against their sources
Stop trusting a clean unload — Ollama's own confirmation gives up after five seconds. Reading Ollama's scheduler source directly: before unloading a model, it polls for the freed memory to actually be reclaimed, checking every 250 milliseconds, for up to five seconds — a fixed constant, not something you can configure — waiting for roughly 75% of that model's estimated memory footprint to be recovered. If that threshold isn't hit before the five seconds run out, it proceeds anyway, logging only that memory usage didn't recover in time. A model disappearing from the running list confirms Ollama's own internal bookkeeping dropped it. It does not confirm the memory has actually been freed. The direct fix is a settle delay after every unload that's actually longer than Ollama's own five-second window, rather than guessing at a "safe enough" pause.
When a model is genuinely wedged, no amount of client-side waiting fixes it — only restarting the server does. Three separate reports on Ollama's own issue tracker describe a model stuck indefinitely in a "stopping" state, confirmed by an Ollama maintainer as the underlying process itself getting wedged. In every case, the only thing that reliably cleared it was killing that process or restarting the service — never a longer wait or a client-side retry, because it's the process that's stuck, not the request. The practical fix for a long benchmark sweep is to restart the serving process itself between models, or every several models, rather than relying only on individual unload requests.
On this specific hardware, no single memory number can be trusted as a hard pass/fail gate. An NVIDIA staff account on the company's own developer forum confirms that the standard GPU-monitoring tool simply cannot report memory usage on this architecture, because unified memory has no dedicated pool for it to report on — the recommended workaround is ordinary system tools instead. Separately, a detailed forum thread on the specific failure mode of the machine becoming an unresponsive "zombie" — SSH dead, no logs written, external monitors still showing the process as alive — traces the cause to a kernel-level accounting gap: the operating system doesn't recognize a large block of GPU-side unified-memory allocation as memory pressure at all, so it never reacts to it the way it would to ordinary memory use. Two different failure surfaces, two different specific causes, the same underlying conclusion: every memory-reporting path on hardware like this is a soft, advisory signal, not something safe to gate a script's behavior on directly.
The freeze isn't always memory — this hardware has a documented thermal cutoff, and it fails exactly as silently. A separately published fix, independently reproduced by a third-party operator running the same hardware, documents the embedded controller performing a hard, unlogged power cut on a thermal spike — a hotspot climbing from roughly 90°C to 98°C in about two seconds, with no software throttling engaging until a 104.8°C critical threshold that the hard cutoff beats to the punch. Before applying a fix, that operator reported roughly twenty of these log-less hard power-offs in three weeks; after capping the GPU's boost clock with a single command, zero. The measured cost was about a 5% drop in throughput. The important part is diagnostic, not just preventive: a silent freeze on this hardware could be memory pressure, or it could be thermal, and after the fact the two look completely identical — no panic message, no error, just a dead machine.
Make every death produce a log line, including the one signal that structurally cannot be caught. A common shell-safety idiom does not catch a background process killed by an external signal — that isn't a bug in how it's used, it's simply outside what that idiom was ever designed to watch. The fix is a trap that fires on any exit and a signal-specific trap that writes a log line before the process actually dies. One hard limit is worth stating plainly, because no amount of scripting works around it: a process cannot trap its own termination if it's killed by the specific signal that Unix reserves as unstoppable. The only way to see that particular kill after the fact is correlating it against the system's own service logs, or decoding a background process's exit status directly — a value of 128 or higher encodes which signal actually did it, information that's already available and mostly goes unused.
Build a completion count that a partially-failed run can't quietly fake. The report-overwrite bug and the mid-benchmark kill that dropped eight of thirteen planned runs share the same shape: whatever silently failed also silently skipped writing an honest account of having failed. The fix is a running tally kept completely outside the process doing the work — know the expected count going in, write one line the moment each item actually finishes rather than one summary at the very end, and let something else compare "lines written" against "items planned," failing loudly the moment those two numbers disagree. A real, similarly-shaped bug on a widely used service manager — a unit reporting a clean deactivation over a process the kernel had actually killed for memory pressure — shows this isn't a one-off risk specific to one script; it's a general failure mode of anything that trusts a process's own account of how it finished.
What actually changes, and in what order
Immediately: add a settle delay after every model unload that's longer than Ollama's own five-second internal window rather than racing against it; restart the serving process itself every few models in a long sweep rather than trusting unload requests alone; and add signal traps plus exit-status decoding to every long-running script, so "died, no idea why" becomes "died, here's the signal." This week: cap the GPU's boost clock and make that persist across reboots, closing off a second, distinct cause of past freezes that a memory-only fix can never touch; and add a per-item heartbeat file with an independent checker watching from outside the process, which is the one mechanism that would have caught every incident from this week's live run before it silently completed short. Next: replace one-off scheduled jobs with proper persistent timer-and-service pairs, which removes the environment-inheritance failure permanently instead of remembering a workaround every time; add a failure hook to every timer-triggered job pointed at one simple local notifier; and build an independent watchdog for the specific signature of a hung-but-technically-alive process — high resident memory, near-zero utilization, elevated CPU — a pattern two unrelated operators of similar hardware have separately reported and that neither a standard health check nor the existing CPU-fallback detector catches on its own.
The throughline across all thirteen incidents is not any single bug. It's that a piece of infrastructure can be unhealthy in a way that every instrument watching it insists is fine — and the actual fix, every time, was never "retry harder." It was making the failure itself impossible, or making it loud.
