About 40 agents on one Strix Halo box: what I learned running three llama.cpp servers on a single iGPU

My agent swarm runs locally. There are 42 registered agents: feed watchers, analysts, judges, a coder, writers. All of them share one mini PC with an AMD Ryzen AI MAX+ 395 (Strix Halo, Radeon 8060S iGPU) and 128 GB of LPDDR5X, of which 64 GB is carved off for the GPU in the BIOS. Cloud models are optional extras. The local endpoint is the only one the system needs.

Here is what I’d tell someone setting up something similar.

The setup

Three llama.cpp servers, one per lane:

• :8000, the workhorse. Qwen3.6-35B-A3B (MoE), one slot, MTP on, reasoning off, 128K context. All long-form generation stays here.
• :8001, the judge fast lane. Gemma 4 12B QAT, two slots, reasoning on. Short yes/no judges and scoring.
• :8002, coding and vision. A 35B model with reasoning on, used for the coder, the large-context judges, and some offloaded planning.

One routing function maps each role to an endpoint. If an endpoint is down, its roles fall back to :8000. That fallback gets used all the time, because sometimes I drop a lane on purpose to free VRAM for something else. That’s a routine trade-off, not a fault, and the routing should treat it that way.

1. Three lanes on one GPU aren’t three GPUs

Each endpoint had its own concurrency semaphore, so a busy lane couldn’t block another one. That sounds right, but it misses the hardware: three servers on one iGPU are time-slicing the same silicon. A long generation on :8000 and another on :8002 can both run at once, and you gain no throughput while you get maximum heat.

The fix is a lease shared across every process: at most one heavy local generation runs at a time. Short fast judges are exempt and stay parallel, because they’re cheap and latency-sensitive. Cloud calls never take the lease. Per-endpoint semaphores answer “is this server busy?”. The lease answers “is the GPU busy?”, which is the question that matters on a single iGPU.

2. A model swap behind an unchanged alias

In August the weights behind :8002 changed to a different model, Qwen3.8. The alias the code routes on didn’t change. Everything kept working: routing resolved, the config read as consistent, and tests stayed green.

That’s the problem. Every threshold tuned on the old model now rested on nothing, and nothing failed to tell me so. Worse, any record stamped with the served model’s alias was now confidently wrong, not merely unverified. If you route on an alias, keep a record of which weights are actually behind it, and treat every calibration as tied to weights, not to the name.

3. reasoning_effort that silently does nothing

The new model’s chat template has a reasoning_effort variable that defaults to xhigh. So the coder and the judges had been running at maximum effort by omission. On a one-line judge prompt at temperature 0 with max_tokens=2000:

• xhigh spent all 2,000 tokens on 8,372 characters of reasoning and returned no answer at all (finish_reason=length, 104 s)
• medium answered in 671 tokens, 52 s
• low answered in 434 tokens, 55 s

Production has a much larger token limit, so nothing was truncated there. But it roughly doubled wall-clock time on a GPU where generations queue behind each other.

The trap is how you set it. A top-level OpenAI-style reasoning_effort field is silently ignored by this template. A bogus top-level value returns HTTP 200. The value has to go in chat_template_kwargs, and a bogus value there returns HTTP 500 with the template’s own exception text. That’s how I verified which path is real. The template also raises on anything outside xhigh, medium and low, so a typo there fails every reasoning call to that endpoint. The other two servers’ templates have no such variable, so the setting must never leak onto them. I now build that payload in one function, with a test that fails if anyone builds it inline elsewhere.

One unresolved worry: at temperature 0, the three levels gave three different verdicts on the same claim (refuted, uncertain, supported). That’s n=1, so it proves nothing yet. But if your judges run on a reasoning model, effort may change the outcome, not just the cost. Measure that before you pick a level.

4. On unified memory, the working set lies

On Strix Halo with Vulkan, GPU allocations come out of shared system RAM and don’t show up in a process’s working set. One measured TTS server showed 2,321 MB of working set against 6,508 MB of commit and 6,184 MB of GPU-local memory.

This matters most in leak hunting. Sample the working set across repeated generations and you’ll see a flat line and conclude “no leak”. But the line is flat because you chose a counter that can’t move for that kind of allocation. Use commit or private bytes, or the GPU process memory counters, and check that your counter can move before you trust a flat reading.

5. Your “optional” cache is maybe not optional

Redis does two jobs here: an LLM response cache, and the event bus the agents talk over (Redis Streams). The cache is optional. The bus isn’t. In one 25-minute outage the fleet didn’t degrade, it stopped: 1,472 connection-refused errors against exactly one unrelated event. It recovered on its own once Redis came back.

Two more Redis lessons from the same box:

• Use volatile-lru, not allkeys-lru, when one Redis is both cache and bus. allkeys-lru can evict stream keys and silently drop events. volatile-lru only evicts keys with a TTL, which is the cache.
• Check your save policy. The old one rewrote a 253 MB snapshot as often as every 60 seconds, up to about 15 GB an hour, onto the system NVMe. That drive reached 98 °C and dropped off the bus. The data directory now lives on a second drive, and the save policy is far less eager.

And on Windows with Docker, a dead Docker engine can leave a socket listening on the port that refuses new connections. “Something is listening on 6379” doesn’t mean Redis is up.

If you are building the same thing

• On a single GPU, guard heavy generation with one lease across all processes, not one semaphore per server.
• Treat every model swap as a recalibration event, even if the alias stays the same.
• Send reasoning_effort through chat_template_kwargs, and verify it took effect by sending a bogus value and checking you get an error.
• On unified memory, measure commit and GPU memory, not working set.
• Write down which of your “optional” dependencies the system can’t actually run without.