On August 27, 2026, an NVIDIA staff moderator wrote in a developer-forum thread about the DGX Spark locking up when it runs out of memory: "I will mark this thread as closed since no one else has reported an OOM hang after the July Update." One week later, on September 3, our own DGX Spark locked up in exactly that way: the machine answered pings, its inference server still answered read-only status requests, but SSH could not complete a login, no new work ran, and the only way back was pulling power. It was running driver 580.173.02, a build stamped June 23, 2026, newer than the 580.159.03 that NVIDIA's release notes list as current. We want to be precise about every side of that. NVIDIA has acknowledged this failure and says it shipped a mitigation in May and July. The thread in question is, as of this writing, still open to replies; the moderator announced a closure that did not happen. The memory over-commit that triggered our freeze was our own misconfiguration, not NVIDIA's. And the "no one else has reported" line was not quite right even on NVIDIA's own forum: two weeks earlier, in the same thread, a user whose machine ran the same driver had described a heavy-load run that "genuinely exhausted the unified-memory pool and took the host down." Our finding is narrower than "NVIDIA is wrong." On at least two units running the same driver, kernel and DGX OS, the machine still hangs rather than failing the process, and the kernel log from our night carries the same driver allocation-failure line that NVIDIA's forum identified months ago.
This piece is the follow-up to our earlier reliability postmortem, which catalogued this freeze as one of thirteen quiet failures and pointed at the NVIDIA thread. What is new here is that we went back and checked everything first-hand: NVIDIA's statements in order, the release notes, the PyTorch tracker, and, most importantly, our Spark's own kernel journal from the night of the freeze, which turned out to have survived on disk. Readers who do not own one of these machines have a reason to care anyway. The DGX Spark shares one pool of memory between its graphics chip and its main processor, the same basic design as Apple's Mac chips and AMD's newest laptop processors, and the industry is moving that way. On a conventional machine, a program that asks for too much graphics memory dies alone. On a shared-memory machine, this story is about what happens instead.
What NVIDIA has said, in order
The original report is a December 6, 2025 thread titled "DGX Spark becomes unresponsive ('zombie') instead of throwing CUDA OOM." A user training a model wrote that "instead of raising a normal CUDA out of memory error and crashing the process, the entire machine becomes unresponsive. SSH hangs, the node stops reacting, but external monitors still show the process 'alive' with no progress. Only a physical reboot recovers it." Two accounts in the forum's NVIDIA employee group engaged within days, asked for reproduction steps, and could not reproduce because the reporter's first recipe accidentally fell back to CPU. No NVIDIA account has posted in that thread since December 10, 2025. It is not marked solved, and its last post, April 10, 2026, is a community member reporting that a common container memory-limit workaround does not work on unified memory because "GPU memory bypasses cgroup accounting entirely" — a 10.7 GB GPU allocation moved the cgroup's own counter by 377 MB. To be fair to NVIDIA, that thread going quiet is not the same as NVIDIA going quiet: the substantive responses moved to a second thread.
That second thread, "Spark hangs - requires a hard-reset," opened January 28, 2026. The same day, NVIDIA moderator aniculescu replied: "This is a known issue which we are actively working to fix. The next major Spark OS release should have better handling of OOM and other issues." NVIDIA's July 2026 release notes then list, under "Improved Memory Management": "The included driver enhances Out-of-Memory (OOM) handling with GB10's unified memory architecture. User feedback is now available when the system encounters memory pressure, which improves robustness and reliability when operating larger models." On August 11, the same moderator wrote that "driver updates released in May and July have mechanisms to stop unit from freezing due to being OOM by stopping processes that take too much memory," and on August 27 wrote the sentence quoted at the top of this piece. The thread's own metadata shows it was never actually closed; it remains open, with that post as its last. And two posts earlier, on August 11, a user running an ASUS-built GB10 machine on DGX OS 7.5.0, kernel 6.17.0-1029-nvidia and driver 580.173.02, the same three versions as our unit, had written that his idle freezes were a different problem (they later turned out to be a Wi-Fi driver fault), but added: "I do also hit the classic OOM variant under heavy load (a 27B model at 256K context genuinely exhausted the unified-memory pool and took the host down — that one fits your description exactly)." He did not date that event. The moderator's reply addressed the Wi-Fi problem and did not engage with that sentence. Separately, a PyTorch tracker issue on the same behavior, opened February 5, was closed April 14 by a PyTorch collaborator who wrote: "We don't see this problem anymore on the latest stack. We tested on canonical kernel version 6.17, pytorch 2.11 and driver versions 580.95 and 580.126."
Two more official documents matter for fairness. NVIDIA's DGX Spark known-issues page has no entry for a hang or freeze, but it does carry detailed guidance that the CUDA memory-query call "does not account for memory that could potentially be reclaimed from SWAP," and tells developers to read the operating system's own memory counters instead, with a debugging workaround of flushing the file cache. That is NVIDIA saying, in effect, that sizing a workload correctly on this machine is the application's job. We agree with that. Our disagreement is only about what happens when an application gets it wrong.
What happened on our unit, from its own journal
Our Spark runs a local research pipeline on Ollama, which is built on llama.cpp, not PyTorch. Two Ollama servers share the machine. On September 3 we shipped a benchmark-derived setting of 48 parallel request slots to that pipeline. The benchmark had been tuned at a 4,096-token context window; production runs 12,288 to 32,768 tokens, and the per-slot working memory (the KV cache) grows with context, so 48 slots at the largest context wanted roughly 130 GiB on a machine with 122 GiB. That was our error, and we said so at the time. What we did not know until this week is what the kernel recorded while it happened, because at the time nothing appeared to be logging at all.
The journal from that boot, which began at 01:04 that morning on driver 580.173.02, DGX OS 7.5.0 and kernel 6.17.0-1029-nvidia, tells the story in five acts. Between 20:33 and 20:38, the NVIDIA kernel driver logged the same line six times: "NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359". At 20:43 the kernel dumped a page-allocation failure for an ordinary background process, and the memory snapshot in that dump shows the file cache down to about 38 MB and under a gigabyte free; the rest of the machine's memory was held by GPU allocations that these counters cannot see. Between 20:50 and 20:53 the Ollama server tried for two minutes to start a model, then gave up: "Load failed ... timed out waiting for llama-server to start." At 21:16 the kernel began reporting tasks stuck in uninterruptible sleep for more than 122 seconds — the network manager, the Wi-Fi supplicant, a kernel worker, then the name-resolution and link-discovery daemons. The last line the journal ever wrote is timestamped 21:21:36. The next entry is the power cycle at 22:40. The lines below are verbatim from that journal, with only the hostname replaced.
Sep 03 20:33:04 spark kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 Sep 03 20:38:32 spark kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 Sep 03 20:43:27 spark kernel: CPU: 18 UID: 0 PID: 2066 Comm: snapd Tainted: G O 6.17.0-1029-nvidia #29-Ubuntu PREEMPT(none) Sep 03 20:43:27 spark kernel: __alloc_pages_slowpath.constprop.0+0x564/0x950 Sep 03 20:43:28 spark kernel: Node 0 active_anon:1086904kB inactive_anon:992852kB active_file:26940kB inactive_file:11736kB ... Sep 03 20:53:04 spark ollama[887746]: level=INFO source=sched.go:641 msg="Load failed" ... error="timed out waiting for llama-server to start: context canceled" Sep 03 21:16:07 spark kernel: INFO: task NetworkManager:1761 blocked for more than 122 seconds. Sep 03 21:16:07 spark kernel: task:NetworkManager state:D stack:0 pid:1761 tgid:1761 ppid:1 Sep 03 21:17:39 spark kernel: INFO: task wpa_supplicant:1770 blocked for more than 245 seconds. Sep 03 21:21:36 spark kernel: nl80211_prepare_wdev_dump+0xe8/0x2b0 [cfg80211] (journal ends; next line is the 22:40 boot)
Three absences in that log are the point. The kernel's own out-of-memory killer never ran; there is no "Killed process" line anywhere. No process was stopped by anything else either; NVIDIA's August description of its mitigation is that it works "by stopping processes that take too much memory," and whatever that mechanism is, there is no trace of it acting. And neither of the two userspace tools the community recommends for this exact situation, systemd-oomd and earlyoom, is installed on our unit. The original December reporter was tuning systemd-oomd on his Spark, so it can be present on this platform; ours reports it as not found, and we do not know whether it was ever there. A 16 GiB swap file was active, which the forum's community advice says to disable. So the honest description of our configuration is: a DGX OS 7.5.0 install with a newer driver, no userspace OOM protection, and a self-inflicted over-commit.
Why we think it is the same bug, and what the log cannot prove
The match to NVIDIA's thread is specific, but it is worth being exact about what it does and does not establish. In March 2026 a community member analyzed the original reporter's NVIDIA bug-report file and wrote that "the kernel log repeatedly shows failures in the driver's internal allocation path: NV_ERR_NO_MEMORY _memdescAllocInternal @ mem_desc.c:1359," that "because this allocation occurs below the CUDA runtime layer, the application may not receive a normal cudaErrorMemoryAllocation," and that afterward a kernel thread "enters uninterruptible sleep (D-state) for more than 122 seconds waiting for a resource that cannot be satisfied." Our journal shows the identical driver line, at the identical source location, followed roughly forty minutes later by identical 122-second D-state warnings. A third user posted the same driver line from their own freeze in April. Two cautions apply. That driver line is the driver's general-purpose allocation-failure log point; it fires on any hard out-of-memory condition, so it proves the driver was refused memory, not by itself that the downstream hang mechanism was the same. And the 122-second figure is simply the kernel's default hung-task timeout. What the log proves beyond dispute is memory exhaustion followed by a hang that the kernel's own killer never interrupted. What it cannot prove is which lock, held by which task, turned exhaustion into a hang.
The differences deserve equal weight. The thread's analysis found the display driver thread stuck; on our unit the stuck tasks are in the wireless networking stack, waiting on a lock inside the kernel's Wi-Fi subsystem, and our log does not identify which task held it. That deserves a direct comparison, because the same thread documents a GB10 Wi-Fi driver fault that mimics a freeze. In that fault the host kept running and only the network dropped: a local logger on the affected machine kept writing normally throughout. Ours was the opposite shape: the network stayed up, pings and read-only HTTP requests kept answering, and it was the host that stopped doing work. That, together with the allocation failures forty minutes earlier and the page-allocation dump showing the machine out of memory, makes starvation the far more likely cause, but we cannot exclude the Wi-Fi driver as a contributor from our log alone. Our workload allocates through llama.cpp's CUDA backend, not PyTorch; we first took that as a possible reason NVIDIA's mitigation might not cover our case, but the other user's heavy-load report was on vLLM, which is built on PyTorch, so the workload difference explains less than we initially allowed. Ours is one unit and one event, theirs is one sentence without a date, and two data points are not a study. We also do not know what NVIDIA's mechanism actually does, whether it lives in the driver or elsewhere, or under what conditions it is meant to engage. Nor has NVIDIA published which driver builds carry the mechanism. Our assumption that 580.173.02 includes it rests on two things: its version number is higher than the 580.159.03 the release notes list as current, and the moderator's August 11 description of the mechanism as already released was written in reply to a user on that same 580.173.02 build. That is an inference, not a confirmation. Any of those could explain the gap, and NVIDIA is the only party that can say which.
One more data point cuts in NVIDIA's favor and ours at the same time. On September 7, four days after the freeze, the same driver logged the same "NV_ERR_NO_MEMORY" line eight more times on our unit, between 20:43 and 20:54, and the machine did not hang. By then our own memory watchdog was running and intervened during that window, unloading models and killing a benchmark at under 8 GB free. So the driver error is not by itself fatal; the hang is what happens at the tail when nothing intervenes. Whether the July driver, our watchdog, or both kept the machine up on September 7, we cannot separate from the outside.
The workaround that has held
The pattern we settled on has three layers, all of which sit in front of the allocation rather than trying to recover after it, because the log above shows there is nothing to recover: once the driver starts failing internal allocations, the machine has minutes left and no remote path back. The first layer is a static budget, checked before the pipeline starts: the number of parallel slots multiplied by the per-slot working memory at the largest context we will ever use, plus the weights of every model that will be resident, must fit inside total memory with a reserve of 25 GiB. The 48-slot setting fails that check; 16 passes. The second layer is a live preflight before any model load, which reads the kernel's available-memory counter, adds what both inference servers already hold, and refuses if the model's weights times a context-dependent factor, with an extra 35 percent margin at the largest context because that exact point has already failed once, plus 12 GiB of headroom, is not free right now. The third layer is a two-minute watchdog on that same counter: below 35 GB free it unloads idle models; below 22 GB it kills any running benchmark outright, on the principle that a lost benchmark is cheap and a 3 a.m. power cycle is not. None of this is new. It is ordinary capacity planning plus the same idea as the community's earlyoom recipe, with application-aware steps (unload idle models before killing anything) and much larger thresholds than earlyoom's default.
The lesson that cost us the freeze was not the missing check but the check on the wrong path. A budget check already existed the night of September 3; it was only wired into the main daemon, and the benchmark that took the machine down entered through a different door. A guard on one path is not a guard. Every entry point now calls the same preflight function.
Since the power cycle on the evening of September 3 the unit has stayed up continuously. The watchdog has intervened 39 times across those four days, 17 of them on September 7 alone while an image-generation pipeline and two inference servers competed for the same pool. Zero freezes. That count is not a boast: a watchdog firing seventeen times in a day means the machine is oversubscribed, and the cure for that is fewer concurrent jobs, not a better watchdog. It does mean the guard held. We would add two caveats for anyone copying this. Our checks trust the kernel's available-memory figure, and NVIDIA's own thread shows that figure can overstate headroom when the file cache is enormous, because reclaiming cache takes time the driver does not wait for; on our unit the cache was tiny at the moment of failure, so the number was honest, but a training workload that streams data through the page cache may need the kernel's memory-pressure metrics as well, as one thread participant suggested. And the community's lighter-weight answer, the earlyoom daemon configured to kill inference processes before the kernel starves, was posted in March, folded into a cluster-management tool within days, and is a reasonable choice for anyone who does not control their workload's admission logic the way we do.
What we have not done
We have not filed this with NVIDIA. That is a public act attached to the operator's identity, and it is the operator's decision; this article is the publicly checkable version of what a report would contain. The other user's report is already in NVIDIA's thread; ours is not. If it is filed, the useful content is the timeline above, the bug-report archive from the affected boot, and the plain question we cannot answer ourselves: what the May and July mechanism does, and whether a driver-level allocation from a llama.cpp process is inside its scope. NVIDIA's engineers have been responsive on these threads, and the fastest resolution here is a second confirmed data point reaching them rather than a debate about whether a thread should have been closed.
For anyone running a DGX Spark unattended today, the practical version is short. Do not trust a memory number as a hard gate on this hardware, not the GPU tool's, not the CUDA query's, and not the kernel's without a margin. Put your admission check on every path that can load a model, not just the one you wrote it for. Run something, earlyoom or your own watchdog, that acts before the kernel starves rather than after. And keep the kernel journal persistent, because ours turned out to have recorded the whole thing while every live instrument said nothing was wrong.
Sources: NVIDIA Developer Forums threads 353752 ("DGX Spark becomes unresponsive ('zombie') instead of throwing CUDA OOM"), 358951 ("Spark hangs - requires a hard-reset"), 362769 ("Mitigating OOM System Freezes on UMA-Based Single-Board Computers") and 365930; NVIDIA DGX Spark Release Notes and Known Issues pages, docs.nvidia.com, last updated August 25, 2026; pytorch/pytorch issue 174358; the affected unit's own systemd journal, driver package history and memory-watchdog log, read September 7, 2026. NVIDIA attributions use the forum's own staff, moderator and NVIDIA-employee group flags; the thread's open state was read from the forum's topic metadata on September 7. This is a report of one unit's behavior, not a benchmark of the platform. Corrected September 8: an earlier version's headline said the thread had been closed, and the text said no other post-July report existed in it.
