llama.cpp pull request 30966 gives the CPU an actual job in mixture-of-experts inference. When the GPU’s expert cache is full and a routed expert isn’t in it, the CPU computes that expert from host RAM instead of waiting around for the upload. The switch is `–moe-cache-cpu-share F`. F runs from 0 to 1, and the default is 0.00, which means off.
The commit states the win condition without any decoration. This is faster when the CPU reads host experts faster than the device can upload them. That happens with fast RAM, a PCIe 4 link, or experts that aren’t pinned in memory. When uploads move pinned experts at full link speed, the CPU share often doesn’t help. That’s it. Commit messages this honest deserve some kind of award.
What Happens on a Cache Miss
Quick background first, because the mechanism only makes sense with it. In a MoE model, each token routes to a handful of experts out of many. llama.cpp keeps recently used experts cached on the GPU. Hits run on the device like always.
Misses used to mean an upload and a wait.
Now the misses split.
With `–moe-cache-cpu-share F`, the device computes the hits and uploads the remaining misses into the cache, while the CPU computes its share of the misses from the host-resident experts. To pull that off, the code builds the expert feed-forward network twice. Once on the device from the cache slots, where misses left to the CPU read zero slots. Once on the CPU from the host experts, where the experts handled on the device get an ID of -1. Then the two results get added together before the down bias.
That ID of -1 is the detail that makes this engineering instead of a hack.
The model can’t tell the split happened. Device partial sum plus CPU partial sum equals the same number one processor would have produced.
One catch before you get excited.
Until the cache is full, every miss uploads anyway. The cache fills at the same speed it would without the CPU share. Cold start doesn’t improve. This flag is about steady state on a warm model, which is exactly the state a long-running local server lives in.
When the CPU Beats the Upload
The commit names three cases where CPU miss processing pays: fast RAM, a PCIe 4 link. And experts not in pinned memory. Read that as one condition wearing three hats.
If what’s slow is moving bytes to the GPU, the feature wins.
If the link isn’t the slow part, it loses.
Pinning is where you’re gonna trip. Host experts are mmap’d by default, and mmap’d experts aren’t pinned. They get pinned with `-lm none`, `-lm mlock`, or `-lm dio`, unless `–no-host` is in play. So the stock configuration is one of the cases where the CPU share helps.
The harder you tune your memory locking, the less the flag does. The commit says it plainly: the device uploads pinned experts at the full speed of the link.
Does the CPU share help your machine? Nobody’s blog post can tell you.
Here’s the trap.
The default setup and the tuned setup want opposite things from this flag. Somebody copies a stranger’s command line out of a forum thread, doesn’t know which memory mode that stranger ran. And ends up with numbers that mean nothing on their own hardware. Which brings us to the other half of the pull request.
The Script That Replaces Guesswork
The same change (PR 30966) ships `moe-cache-tune.py`, and I’d argue it matters as much as the flag itself. The script fits the upload and CPU speeds of your machine from real runs, then replays the routing from all those runs through a copy of the cache logic to score other cache sizes and shares.
Think about what that replaces. The old way to tune a local model was to try values, watch tokens per second, and half-remember what worked. The commit’s own guidance says the best share depends on the machine, the model, and the cache size. Translation: nobody’s numbers transfer to your rig.
A script that measures your hardware and simulates the alternatives turns folklore into arithmetic. And that’s the pattern I want more of in local inference tooling.
My read is simple.
The flagship feature will help some machines and do nothing on others.
And the maintainers knew that when they wrote the commit. Shipping the measurement tool next to the feature is an admission that the answer is machine-specific. I trust software that brings its own disclaimer.
What To Actually Run
Start from the constraint, not the flag.
A widely cited tuning guide puts it in one line: “CPU expert offload is a VRAM-saving tool, not a performance tool.” The recommended first move is full GPU residency when VRAM allows, using `-ngl 99 -ncmoe 0`. Whole model fits on the card? Keep it there and close the flag documentation.
Once it doesn’t fit, the decision tree earns its keep. The older flag, `–n-cpu-moe N`, keeps MoE weights from the first N layers in CPU memory, per the server README. The new cache flag then handles what happens when a running model keeps asking for experts that aren’t on the GPU. Together they let a bigger MoE model run on the card you already own, at whatever speed your RAM and PCIe link negotiate.
For a small operator running local models for real work, this is the actual story.
Local inference isn’t limited by model size anymore. It’s limited by data movement. Your GPU has compute to spare and a memory bus that doesn’t scale with it. Every optimization worth having is a decision about which processor reads which bytes fastest. And that’s a question you settle with measurements, not with a benchmark somebody else posted.
So before you spend money on a bigger card, spend an hour on the one you have.
Pull the latest llama.cpp and try full residency with `-ngl 99 -ncmoe 0` first.
If the model doesn’t fit, run `moe-cache-tune.py` and set the cache size and CPU share it recommends for your hardware. Write down what it reports. If you’re serving local models to anything that matters, those numbers are worth more than every hot take about which GPU to buy next. Including mine.
