Tuning an NVIDIA DGX Spark (GB10) to serve many concurrent local LLM agents

Measured September 2026. Every number here came off one machine running the workload it describes — dozens of small agents reading web pages in parallel — not from a spec sheet.

Before you copy a concurrency number out of the tables below, read section 3. Every measurement here was taken at a 4096-token context. Our real workload runs at 12288–32768, and KV cache scales with context. The slot count these tables appear to bless needed more memory than the machine physically had, and hung it. The throughput-versus-concurrency shape is real; the slot count is only valid at the context it was measured at.

The DGX Spark is new enough that when we sent several hundred local agents to search the web for exactly this information, they came back with nothing usable. So here is ours.

Hardware: NVIDIA GB10 Grace-Blackwell, 122 GiB unified memory, aarch64 Linux, CUDA 13. Software: Ollama 0.33.2, flash attention on, KV cache quantized to q8_0. Models: qwen2.5:7b (Q4_K_M) as the parallel worker, qwen3:30b-a3b (MoE, ~3B active) as the planner and synthesizer.

The headline numbers

Both models resident, OLLAMA_MAX_LOADED_MODELS=2, num_ctx 4096 — see the warning above before reading a slot count off this table:

One model resident at a time, pushing further:

(All at 4096-token context. At 32768 these slot counts do not fit — see section 3.)

For scale, a 72B dense model measured 4.4 tok/s on this same machine. That is why it is not in the table.

Five things worth knowing before you tune this box

1. The cgroup cannot see unified memory. Do not trust MemoryMax.

This is the finding that cost us the most and that we have seen written nowhere else.

If you run inference under a systemd unit and watch MemoryCurrent, you will get a number that has almost nothing to do with reality. Ours read 2.1 GB while nvidia-smi showed that same unit's llama-server children holding 25 GB. On unified-memory hardware the weights and KV cache are not accounted as the cgroup's anonymous memory.

The consequence is worse than a wrong reading: a memory watchdog written against MemoryMax will never fire. Ours did not, and we thought we were protected for a day.

Measure MemTotal − MemAvailable from /proc/meminfo instead, and cross-check with nvidia-smi --query-compute-apps=used_memory.

2. Raise the server's parallelism before you raise the client's

Our first benchmark varied client concurrency — 1, 4, 8, 12 — against a server left at OLLAMA_NUM_PARALLEL=8. Throughput fell at 12, and we published "8 is the sweet spot."

It was nonsense. Every request past the eighth was sitting in a queue inside the server. We had measured our own configuration and mistaken it for the hardware. The experiment could not have produced any other answer.

Set OLLAMA_NUM_PARALLEL first, confirm it took effect (systemctl show <unit> -p Environment), then vary the client. When we did that, the real figure was not 8. It was at least 64, and still climbing.

3. "Slots are nearly free" is a trap. Benchmark at your real context.

Look at the memory column in the tables above: 40.3 GB at 8 concurrent and 40.3 GB at 32 for the 7B. 56.9 → 65.4 GB across 8 → 64 for the MoE. Adding concurrency appeared to cost almost nothing in memory.

We wrote that down, set 48 slots, and hung the machine within the hour.

Those benchmarks ran at num_ctx 4096. The production workload runs at 12288 for page readers and 32768 for synthesis. KV cache scales linearly with context, so the same 48 slots need:

Throughput still climbs with concurrency, and the shape of that curve is real. The slot count is only valid at the context it was measured at.

What "hung" looked like, because the symptoms are not what you would expect. ICMP kept replying. The inference server's /api/tags answered HTTP 200 with a full model list. But /api/generate never returned, and sshd accepted TCP connections and then never sent its banner. Anything already resident kept working; anything needing a new allocation or a new process blocked forever. No OOM kill, no error in any log we could reach, no way in. It took a power cycle.

So: benchmark at the context you will actually run, and treat concurrency as a memory budget you compute before you set it, rather than a number you find by turning it up. We now refuse to start a configuration that cannot fit at our largest context:

48 slots at 32768 ctx needs ~128 GiB of 122 GiB  ->  refuse to start
16 slots at 32768 ctx needs  ~73 GiB of 122 GiB  ->  fine

A runtime memory watchdog does not save you here, and the reason is worth knowing: the server allocates KV for every slot in one step when a model loads. By the time free memory moves, the allocation has already happened. Admission control is the wrong tool. A static budget, checked before startup, is the right one.

4. A thinking model under format: json can return nothing at all

qwen3-class models will spend their entire num_predict budget inside a reasoning block and hand back an empty content field. Our first full-depth run died at "planning produced nothing" for exactly this reason, and the error surfaced as a JSON parse failure on an empty string — several layers away from the cause.

Pass think: false. But check first: a model without the capability rejects the parameter outright. Ask /api/show for the model's capabilities and only send think when thinking is listed.

Also: when a JSON generation comes back with done_reason: "length", it was truncated mid-structure and will never parse. Retry with a larger budget rather than discarding it.

5. Install a second Ollama rather than upgrading the one you have

We needed a newer Ollama than the one already serving half a dozen other things on this box. Rather than upgrade in place and put all of them at risk for the benefit of one workload, we extracted the new release to /opt/<name>/ and pointed a single systemd drop-in at it:

[Service]
ExecStart=
ExecStart=/opt/<name>/bin/ollama serve
Environment="LD_LIBRARY_PATH=/opt/<name>/lib/ollama"
Environment="OLLAMA_LIBRARY_PATH=/opt/<name>/lib/ollama"

The system-wide binary stays where it is. Rollback is deleting one file.

Verify the GPU is actually being used before you trust it. A generic aarch64 build without kernels for your compute capability will silently fall back to CPU, which looks like success and runs dozens of times slower. Generate a few tokens and check nvidia-smi --query-compute-apps lists a process from the new path.

Reproducing this

Nothing here needs our code. For each concurrency level, fire N identical chat requests concurrently against /api/chat with stream: false, sum eval_count across the responses and divide by wall-clock time for the aggregate rate. Pin num_ctx explicitly — a large default context will size your KV cache for you and the memory figures stop meaning anything.

Sample MemAvailable once a second throughout, and abort before the next level if it drops below a floor you choose in advance. On unified memory an out-of-memory event can take the whole machine down rather than raising an error in one process.

Caveats

• One machine, one afternoon, two models. Treat these as a starting point for your own measurement, not as constants.
• Everything was measured with q8_0 KV cache and flash attention on. Other settings will move the memory numbers.
• The workload is batch extraction and summarization, where per-stream latency does not matter. If you are serving an interactive chat, the high-concurrency rows are the wrong end of the trade.
• Ollama 0.33.2. Newer releases may change all of this.