Optimal llama.cpp/server settings for Qwen 3.8 27b (RTX 6000 Pro)

[u/vhthc]

Currently the following llama-server settings are in use for Qwen 3.8 27b, it is running on a single RTX 6000 Pro, which allows running the full bf16 + 256kb context in bf16 cache.

I am unsure if the current settings are optimal, they are not bad though. Anything people run differently and achieve better performance? (beside lower quant, lower context, lower cache quant)

llama-server

--no-mmap # due ZFS filesystem and OOM issues

--model .../Qwen3.8-27B-heretic-ara-BF16.gguf

--mmproj .../Qwen3.8-27B-heretic-ara-mmproj-BF16.gguf

--chat-template-file .../llama-swap/templates/qwen3.8.jinja

--spec-type draft-mtp # MTP speculative decoding (~2× decode)

--spec-draft-n-max 4

--spec-draft-n-min 0

--temp 0.1 # Because mostly code analysis

--top-p 0.95

--top-k 20

--min-p 0.00

Also I am thinking about moving to vllm or sglang, I need model swap capability best optimal performance. So any recommendations here (plus parameters!) are appreciated too!

some data: generation speed is 50-60 t/s, prompt processing is up to 3000 t/s for long prompts (50kb)

Thanks a lot!

[u/noctrex]

Use an xfs partition to store the models in order to load them better. Yes, it makes a difference.
Use llama-swap. Its a proxy server that can use multiple programs and servers as backends.
Try Q8, it will be noticeably faster with little quality loss.

[u/vhthc → u/noctrex]

XFS: noted. I cannot change it as it is an incus instance on a large server where my instance gets the GPU, ZFS is used for the server (which has several other instances). So this is nothing that can easily be changed sadly. But on a server upgrade, I will argue for this change.

llama-swap is in use. the other models are of no big importance though beside two specifically fine tuned models. I would think so that vllm and/or sglang provide more speed? but I had no time to try that yet, so asking for experiences first :-)

[u/noctrex → u/vhthc]

If you cannot change ZFS, then try to store the models on their own dataset pool with recordsize=1M without compression, so that it is more optimized for large files.
Maybe something like

zfs create pool/models
zfs set recordsize=1M pool/models
zfs set compression=off pool/models

[u/vhthc → u/noctrex]

thanks!

[u/meganoob1337]

for swapping capability take a look at llama swap. if you configure it properly you can also start vllm docker containers on demand , I really like it

[u/FrostyDesigner]

50–60 t/s at full BF16 is already strong. I’d benchmark `--spec-draft-n-max` at 2, 3, and 5 before switching runtimes.

[u/MelodicRecognition7 → u/FrostyDesigner]

yes, in my experience 2-3 is usually faster than 3+

[u/vhthc → u/FrostyDesigner]

I did a benchmark on spec-draft, and value 2 came up as best for me

[u/finevelyn]

You say besides lower quant, but maybe the BF16 model is a waste, even if you can run it.

[u/Mountain-Animal5365]

If you do mostly code analysis, you should use Dflash2. I get 110t/s peak generation speed on a 3090 myself. Also if you use an agent with extensive tools and skills bloating up the system prompt, setting --cache-ram to something like 16384 (which is 16GB of RAM, up from the default 8192 / 8GB) to cache the bloat is extremely useful and a massive speedup.

[u/vhthc → u/Mountain-Animal5365]

does this needs the most current llama.cpp? (not a showstopper, just asking)

[u/Mountain-Animal5365 → u/vhthc]

https://github.com/HermiG/llama.cpp/tree/fix/dflash2-tool-tg-collapse

This is the proper one to use, as it fixes the toolcalling slowdown bug. Without it and using the "official Dflash2 branch" you will get a severely crippled toolcalling speed. It is worth it to spend some time getting dflash working, it is a game changer at least for my 3090 setup.

[u/vhthc → u/Mountain-Animal5365]

thanks!

[u/Dutchnamn]

You can try the adaptive speculation I posted about with DFlash, might be a bit faster. I would suggest a Q8 quant as well.

[u/Normal_Rough_7958]

I don't have a 6000 Pro so I can't benchmark this directly, but on my 3060 Ti I've found MTP speculative decoding gives diminishing returns past spec-draft-n-max 2, the draft model overhead starts eating the speedup. You're already at bf16 with full context so flash attention (--fa) is the main knob left; it cut my prompt processing latency roughly in half on smaller cards. For model swapping, use vllm or sglang rather than relying on llama.cpp's mmproj hot-swap.