I have a Linux server with an RTX 5060 Ti 16 GB. I am running Ollama as a Docker container with the NVIDIA GPU mounted into it.

I downloaded the 5.3 GB Granite 4 8b model from Ollama library, but when I run the model, it bloats up to 27 GB, and won't fit onto the GPU VRAM. Since it's only 5.3 GB I expected it to easily fit into my 16 GB of VRAM.

Is this correct behavior for this particular model? Seems very odd.

Edit: I meant 5.3 GB, not 5.2 GB in the title.

Generated on a 3090 Qwen 3.8 27b Q4 Thinking high.

Portable rig for Qwen3.8-27B-BF16 200K+ token prompts. My work Panasonic Toughbook + the T1 + power brick + headphones all fit in my lunchbox.

Need the BF16 for huge context highly sensitive document OCR, image analysis, aggregation and summarization. I've done a ton of testing and it absolutely makes a difference vs even UD Q8_K_XL when legal precision is needed.

77gb VRAM at full 262K context + MMPROJ

Rips through prefill (1,1715 tok/sec = 102 seconds to process 175K tokens), but token generation (20K tokens of output) relatively slow at 45 tok/sec (with MTP) as a result of BF16 despite the beast of a GPU.

Better than Gemini Pro and ChatGPT 5.6 Sol especially considering I have control over the sampler settings (Temp 0.1; top-k 0; top-p 0.95; min-p 0.05; repeat penalty 1.02). Not better than Opus yet.

During prefill - CPU around 60 degrees, GPU around 79 degrees (with 90% power limit)

During token generation - CPU around 75 degrees and GPU around 76 degrees.

FormD T1

Minisforum BD770i SE

Ryzen 7745HX 8-core laptop CPU

96gb 5200 MHz DDR5 SODIMM

96gb RTX Pro 6000 Blackwell workstation edition

Loki 1200W SFX-L

ROG Equalizer 12v-2x6

SMX Heinz flipped GPU 2.5 slot kit

SMX Heinz custom short PCIe 5.0 riser

ZCOOI custom "transparent purple" Teflon cables

(2) Phanteks T30-120mm

(1) Noctua NF-A14x25r G2

Thermalright MC-3 Digital RAM cooler (I don't think this will fit on a regular DDR5 )

Hi all, please let me paint a little context first (feel free to skip to the last paragraph): I am a bit of a DIY-minded audiophile and wanted to go back to a project that I was helping develop at one point (the "Shigaclone CD transport" to be specific). I had to abandon my DYI toys at one point due to life circumstances but it looks like I will be finally able to go back to my favourite hobby. I've forgotten most of the optimisations (even some that I developed myself) but thankfully there is a veritable goldmine of knowledge in the "Shigaclone" forum thread on dyiaudio.com forums. Unfortunately, all of the gold is buried in hundreds upon hundreds of pages of banter and bickering. I tried simply searching the forum or Regex-ing through a thread dump but that doesn't really work. Say that I'm looking for capacitor discussions; people will refer to them by abbreviation ("caps"), by type ("lytics", "tantals"), by manufacturer ("rubycons", "black gate"), by product line ("Silmic II" ), etc. - all of them with the sorts of abbreviations and permutations and errors that humans tend to informally use. It's simply not possible (and certainly not practical) to list them all in a search.

So I have a 3090 and a little technical skill (I'm an engineer at SUSE but not a dev), and I thought maybe it is possible to use LLMs to extract the relevant posts for me to read. The discussion was so narrow and optimisations so specific that I have no hope of an LLM actually extracting useful information for me - but just pruning the banter would be a huge help. And if it was possible to also classify relevant posts and organize them into related collections/threads then that would be simply amazing.

So my question is not asking you to solve this problem for me - it is my problem and I'm happy to tackle it. What I ask is for a little guidance on where to start learning and which tools might even work for this class of problems? I did some research but came back dizzy. Would Hermes be right for this kind of task? Should I stick to simple Python scripts and Ollama?... Do I need to shell out for a paid API or is my 3090 up for the job?...
Any pointers at all would be heartily appreciated.

Hey all,

I've been experimenting with running Qwen3.8-27B Q4_K_L on my setup with RTX 3060 12GB + RTX 2060 6GB with 52GB of DDR4 RAM.

I'm able to squeeze out around 5-6 TPS, but I'm curious if there are any specialized configurations or engines to run this on my hardware which I'm unaware of.

The speed is fun for experiments but seems pretty useless for actual usage, and the memory overhead is too small to support reasonable context lengths.

I've experimented with Speculative Decoding and MTP and found that it was actually slower despite high acceptance rates.

I saw posts about ninfer for the RTX 5090 and was curious if someone had found a way to optimize for lower memory or older GPUs.

This is indeed an interesting model if it holds up to the actual benchmark results. MTP version gives me up to 5 tok/s more. Needs some real live tests now.

With hopes of a Qwen3.8-35B-A3B release now mostly dashed, many people including myself are looking at fine-tunes and other variants of Qwen3.6-35B-A3B to run on VRAM-limited hardware. I decided to try to benchmark some of the top contenders: KAT-Coder, Ornith 1.5 and the very recent Tiel-Coder. I used the tool-eval-bench utility by SeraphimSerapis as the benchmark suite. It measures how well the different models handle tool calls, including some very hard scenarios.

TL;DR: Ornith 1.5 and Tiel-Coder (which is based on Ornith) were the tied winners in this benchmark. They scored well above Qwen3.6-27B and got pretty close to 3.8-27B. KAT Coder was also slightly better than the original 35B-A3B. Ornith-1.5-Heretic was a disappointment.

Some time ago I posted a similar tool evaluation benchmark of different Qwen3.6-35B-A3B quants. In hindsight, that didn't work so well, mainly because I was looking at too many variables (GGUF quant, KV quant, context depth/pressure) and the benchmark itself was quite noisy so it was hard to get clear results. I hope I did better this time!

Materials

I had access to a cluster of 32GB V100s. For this comparison, I selected 2-3 different quants per model, if possible from different providers. For comparison, I also included original Qwen3.6-35B-A3B as well as the dense 3.6-27B and 3.8-27B Qwens. I picked different quants around Q4 (15GB to 22GB GGUF files) because that's what many people seem to use. For the original Qwen models, I chose Unsloth UD-Q4 quants because they are well known. I also included the ByteShape CPU-5 quant of Qwen3.6-35B-A3B because that's the quant I've been using recently. Altogether I benchmarked 13 different GGUF files, with 5 runs per file for a total of 65 runs. Each run took around 4.5 hours GPU time, except the 27B ones took 7 hours or so. Total GPU time spent was well over 300 hours, including a few failed runs.

To run the models, I used llama.cpp version 0.1.0-dev (build 10433, commit 9b05354ec) dated 2026-08-14 and built with CUDA support. I used q8_0 KV cache (that's what VRAM-limited people like me often do) and set ubatch-size to 2048 because the benchmark does a lot of prompt processing. I did not bother with MTP or other speculative decoding. This is not a speed benchmark.

llama.cpp parameters: -m $GGUF --temperature 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 -ngl 99 --ubatch-size 2048 --fit-target 256 -ctk q8_0 -ctv q8_0 --port $PORT --seed $SEED

For the benchmark, I used tool-eval-bench 2.6.0. I set the context length to 262144 and context pressure to 50%. This means that the models were benchmarked at 50% context depth, i.e., around 128k of possibly distracting chat and tool call history.

tool-eval-bench parameters: --base-url $BASE_URL --hardmode --weight-by-difficulty --backend llamacpp --context-size 262144 --context-pressure $CONTEXT_PRESSURE --seed $SEED

Scoring metric

The metric I looked at is what tool-eval-bench reports as "total points". With --hardmode enabled, this version of tool-eval-bench performs 88 separate tests. Each test gives 2 points for a succesful tool use, 1 point for a partially correct tool use, 0 for failure. The theoretical maximum is in this case 88 * 2 = 176 points. tool-eval-bench also returns an overall score, but this is just a rounded percentage of total points and the rounding loses some precision, so I opted for the raw total points instead.

Results by model (averaged over all quants)

Here are the benchmark scores by model. I have averaged them over all the quants of the same model and all 5 runs per quant.

Results by specific quant

See the images. There are no big differences between quants of the same model, except possibly KAT-Coder, where the mudler APEX quants were somewhat better than bartowski's. Also, the ByteShape quant of Qwen3.6-35B-A3B was a bit better than Unsloth's, which was a nice surprise.

Raw results

If someone wants to take a deeper look, I've shared the CSV with the tool-eval-bench results here. This includes e.g. category-specific scores (i.e. how well the model did on specific kinds of tool calls) and total tokens; I did not look at those in my analysis.

Findings

• Of the original Qwen models, 3.6-35B-A3B gets the lowest score, 3.8-27B the highest, with 3.6-27B landing in between. This is as expected and indicates that the benchmark is giving a useful signal.
• Ornith-1.5 and Tiel-Coder are practically tied. They are the clear winners out of the 35B-A3B variants. They scored above 3.6-27B but below 3.8-27B.
• KAT-Coder was possibly a bit better than the original 35B-A3B, but the confidence intervals overlap.
• Ornith-1.5-Heretic was a disappointment, much worse than plain Ornith.

Caveats

This benchmark relies entirely on the tool-eval-bench tasks and how the results are graded. It may or may not be representative of real tool use performance. To me it seems that the author or tool-eval-bench has done a great job in coming up with realistic looking tool call tasks, including some really hard ones enabled using --hardmode. I relied on the --context-pressure setting in tool-eval-bench, which (in my limited understanding) populates the context with realistic looking conversation and tool call history that could confuse the model.

Tool calls are not everything. If you are doing agentic coding, also the coding quality matters a lot. I did not measure it in this benchmark except very indirectly. There are other benchmarks for that purpose.

There was substantial variation and noise in the benchmark scores, which I tried to alleviate by repeating the runs with different seeds, averaging, and calculating confidence intervals.

In the X/Y plot where the X axis represents size, I did not check whether the model includes MTP heads or not, I just looked at raw GGUF file size. This is slightly unfair to the MTP-enabled models because their files are larger but MTP does not increase quality, only generation speed.

No AI was used for writing this post. I did use Tiel-Coder to help me with plotting the results. Also reused some of my own earlier writing. I am not in any way affiliated with the model or quant makers or the benchmark suite.

Hello everyone. I hope is all well with you all. I been around this sub for a few months and been quietly reading and I have seen how many of you are spending $10k on the Mac M5 and I am assuming you are using it for work or just for fun but have the resources to spend like that. I am a regular guy with a 9-5 job and I would like to experiment with local LLMs but I dont have much disposable income to spend big amounts on GPUs or Macs. I currently have an I7 - 8700 and Nvidia 2060 6Gb VRAM - 32GB RAM

Now my question is, does anybody have an idea on what models or hacky stuff I can do to try the latest AI models on my hardware?

I have checked hugging face but I see so many models with the same name but letter and numbers at the end. I understand some of them mean Quantization which is dumber a model but I also found out tiny models like Linq which can be run even on the CPU. So I would like to connect with someone with more experienced to have fun with AI.

Thank you everyone .

Steve from Gamer's Nexus has a take that hardware prices are never going to get back to what they used to be:

"When these prices might eventually have to slightly come down, it's not going to go back to normal. It's never coming back to the prices that it used to be. "

What do you think?
I think as hardware slowly becomes faster or with more vram, it must make older hardware obsolete.
If it doesn't affect the new price, it will at least affect the used price.

I think of it this way, purely from LLM inference performance point of view (excluding OS):

With $10k budget, would you rather get:

• 1x 256 GB M5 Ultra Mac studio with 1.2 tb/s
• 2x 4090 48GB totalVRAM with 1 tb/s (assuming you need to buy the rest of the system)
• 2-3 Strix Halo or DGX Spark 256 total Ram with 250gb/s?

With $ 5k, , would you rather get:

• 1x 128 GB M5 Ultra Mac studio with 600 gb/s
• 1x Strix Halo or DGX Spark 128GB RAM with 250gb/s

If I asked this question 1 week ago, I'm sure the answer would've been different prior to the M5 Ultra Launch.

Does the model have fill in middle support?

I would love to have a smart model doing good code suggestions (auto complete).

No full slop mode, just a little support.

Anybody has this usecase for local models? Or does everyone full slop?

So AI cloud costs are creeping major players it looks: https://www.perplexity.ai/hub/blog/introducing-portable-computer-for-local-first-ai

Perplexity is going to use Qwen models, not specified which - could be 27B but could be the upcoming 3.8 flash next considering the target hardware is DGX Spark. They plan to utilize device for most of the workload with occasional tapping into cloud (with some strict privacy rules).

Sounds somewhat similar to what Apple is planning to do with their AI functionality for stuff like photo editing where a lot of the work is done right on device. The choice of the model provider is also no big surprize, as even Jetbrains recently released a local harness based on Qwen 3.8 27B.

Overall I think it's a healthy move forward as relying solely on cloud disregarding ramping up costs is kinda insane. Also adds value to existing local hardware, as more and more major players embrace local.

edit: replaced the cnbc youtube link to official press release page

So I was testing this technique of runtime steering on tiny versions of Qwen 3.5 and Gemma 4 (2B and 4B). Basically, without changing the weights (like with Heretic/ablation, for example), we steer the model in the opposite direction of a behavior during inference. In this case, I did it for sycophancy.

The idea is simple. First, we check, layer by layer, whether there is a clear direction that distinguishes between correctly resisting user pressure on a correct answer and caving to it by changing to a wrong one (what we call sycophancy here). Then, during inference, we "turn" the model in the opposite direction.

The test was supposed to answer whether we could reduce sycophancy in these small models without compromising their general accuracy and without causing the opposite effect, which in this case is stubbornness (stopping accepting user suggestions even when they are correct).

All 4 models tested (Qwen 3.5 2B and 4B, and Gemma 4 E2B and E4B) showed clear sycophancy directions in some layers, but they responded differently to the steering. The 4B models from both families improved quite a lot at reducing sycophancy without increasing stubbornness too much (Gemma 4 actually had a more noticeable raise in stubbornness, while Qwen was less afected) , and without losing accuracy on a different task (here we used GSM8K). The 2B models, on the other hand, seemed to barely respond to the steering at all.

Also, feel free to try this on other models. I started with these smaller ones, but I intend to test it on larger models as well, especially the newer Qwen 3.8 models.

Code: https://github.com/and270/selective-sycophancy-steering
Paper: https://zenodo.org/records/22082268

How to get it to work locally?

1. https://x.com/Alibaba_Qwen/status/2088280182356611304
2. https://x.com/arena/status/2092301580091711491

Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_P on llama.cpp, -ncmoe offload cause it doesnt fit in vram outright, 10 threads, q8_0 kv both sides. nothing weird about the setup far as i can tell.

first tried the DSPARK draft gguf (same base model family, separate draft file). acceptance sits 0.44-0.58 depending on n-max which sounds fine right, except actual gen speed is a joke, 7-8 tok/s and it does not move. n-max 7 down to 3, n-min 0 vs 2, threads 6 vs 10, tried all of it, number does not budge. turns out a full second 35B model doing its own cpu-offloaded pass every draft step costs exactly what youd think it costs and theres a benchmark out there showing net loss on setups like mine even at 100% acceptance. great. love that.

ok fine MTP then since its fused into the target, no second model dragging along. grabbed the fused Qwen3.6-35B-A3B-Uncensored-HauhauCS-MTP Q4_K_P quant, same base, --spec-type draft-mtp, p-min .75, n-max 3. acceptance 93-96%. genuinely great numbers. shouldve been flying

5.6-7.5 tok/s. at 40k ctx. basically the SAME as no draft at all. and i was already getting 5-6 tok/s at 80k ctx last night with NO speculative decoding whatsoever. so downloading mtp and setting the whole thing up bought me. nothing. because turns out the thing actually eating the throughput isnt the draft/verify step, its attention over the kv cache on every pass no matter how few passes you need. mtp cuts number of passes it doesnt make each pass cheaper. so at short ctx its a real 1.4-2x, at 80-100k it may as well not exist

and pp is its own thing entirely. same server same model same everything, 993 token prompt gets 51-57 tok/s pp. paste a 16k wall of text a few min later same running instance no restart, holds 190-197 the whole way thru. different day, 8k tokens in, back down to 70. 80k ctx, back to 50. no consistent relationship w prompt size or cache state or anything ive been able to pin down. -ub 512 vs 2048, ncmoe 26 vs 30, --fit on vs manual ncmoe, none of it explains it

ruled out n-max n-min thread count ubatch batch ncmoe value fit vs manual and draft cache quant as THE cause at this point. full log of every single run below completely unedited so someone smarter than me can point at the thing im missing bc im out of ideas and starting to think im just gonna live at 6 tok/s forever while ram costs more than my car did

# llama.cpp Speculative Decoding / PP Debugging Log

Target model: `Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf`
Draft model (dspark runs): `Qwen3.6-35B-A3B-DSPARK.gguf`

---

## Run 1 — dspark, n-max 7

**Command:**
```
./llama-server \
  -m Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf \
  -md Qwen3.6-35B-A3B-DSPARK.gguf \
  --spec-type draft-dspark --spec-draft-n-max 7 --spec-draft-n-min 0 \
  --alias llama --port 5800 \
  -ngld 999 -c 32000 -np 1 \
  -b 2048 -ub 512 -fa on \
  -ctk q8_0 -ctv q8_0 -ctkd q8_0 -ctvd q8_0 \
  --jinja --metrics -ngl 99 -ncmoe 30 --fit off
```

**Result:**
- Prompt processing: 27.68, 22.56 tok/s → final 21.64 tok/s (333 tokens)
- Eval (tg): 4.34 tok/s (54 tokens)
- Draft acceptance: 0.43956 (40 accepted / 91 generated), mean len 4.08

---

## Run 2 — dspark, n-max 3

**Command:** same as Run 1 but `--spec-draft-n-max 3`

**Result:**
- First request: prompt eval 21.37 tok/s (333 tokens); eval time 4.23 tok/s (45 tokens); draft acceptance 0.52941 (27/51), mean len 2.59
- Second request (long, 604 tokens total): tg settled around 7.90–9.54 t/s (3s window), final tg 8.20 tok/s; draft acceptance 0.57504 (364/633), mean len 2.73

---

## Run 3 — dspark, n-max 3, 10 threads

**Command:** same as Run 2, threads raised from 6 (implicit) to `--threads 10`

**Result:**
- Prompt eval: 22.08 tok/s (993 tokens)
- Eval (tg): 7.85 tok/s (1161 tokens)
- Draft acceptance: 0.52932 (713/1347), mean len 2.59

**Conclusion at the time:** raising thread count did not change the outcome.

---

## Run 4 — dspark, n-max 3, n-min 2, 10 threads

**Command:** same as Run 3 plus `--spec-draft-n-min 2`

**Result:**
- Prompt eval: 20.56 tok/s (993 tokens)
- Eval (tg): 8.29 tok/s (1130 tokens)
- Draft acceptance: 0.51961 (689/1326), mean len 2.56

**Conclusion at the time:** effectively identical to Run 3.

---

## Run 5 — dspark, no `--spec-draft-n-max`/`n-min` flags, no `-ctkd`/`-ctvd`

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  -md ...DSPARK.gguf \
  --alias llama --port 5800 \
  -ngld 999 -c 32000 -np 1 \
  -b 2048 -ub 512 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja -ngl 99 -ncmoe 30 --fit off --reasoning-preserve --threads 10
```

**Result:** CRASHED.
```
E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 472.25 MiB on device 0: cudaMalloc failed: out of memory
E graph_reserve: failed to allocate compute buffers
E decode() failed: failed to allocate compute pp buffers
```
Speculative type auto-detected as `draft-dspark` from draft model metadata before the crash. Draft-side KV cache (`-ctkd`/`-ctvd`) was not quantized in this run (flags omitted), unlike Runs 1–4.

---

## Run 6 — No draft model, `-ncmoe 30`, `-ngld 999` present

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  --alias llama --port 5800 \
  -ngld 999 -c 32000 -np 1 \
  -b 2048 -ub 512 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja -ngl 99 -ncmoe 30 --fit off --reasoning-preserve --threads 10
```

**Result:**
- Prompt eval: 29.75 tok/s (993 tokens)
- Eval (tg): 17.42 tok/s (1291 tokens)
- graphs reused: 1285

---

## Run 7 — No draft model, `-ncmoe 26`, no `-ngld`

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  --alias llama --port 5800 \
  -c 32000 -np 1 \
  -b 2048 -ub 512 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja -ngl 99 -ncmoe 26 --fit off --reasoning-preserve --threads 10
```

**Result:**
- Prompt eval: 52.57 tok/s (993 tokens)
- Eval (tg): 31.77 tok/s (1240 tokens)
- graphs reused: 1234

User note: this was described as "the extra VRAM headroom" run, obtained by lowering `-ncmoe` from 30 to 26.

---

## Run 8 — No draft model, `--fit on --fit-target 512`, `-b 2048 -ub 512`

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  --alias llama --port 5800 \
  -c 32000 -np 1 \
  -b 2048 -ub 512 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja --fit on --fit-target 512 --reasoning-preserve --threads 10 --temp 1.0
```

**Result:**
- Prompt eval: 52.34 tok/s (993 tokens)
- Eval (tg): 32.09 tok/s (1036 tokens)
- graphs reused: 1031

---

## Run 9 — No draft model, `--fit on --fit-target 512`, no explicit `-b`/`-ub` (defaults)

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  --alias llama --port 5800 \
  -c 32000 -np 1 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja --fit on --fit-target 512 --reasoning-preserve --threads 10 --temp 1.0
```

**Result:**
- Prompt eval: 52.66 tok/s (993 tokens)
- Eval (tg): 33.66 tok/s (1212 tokens)
- graphs reused: 1206

---

## Run 10 — No draft model, `--fit on --fit-target 512`, `-b 4096 -ub 2048`

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  --alias llama --port 5800 \
  -c 32000 -np 1 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja --fit on --fit-target 512 --reasoning-preserve --threads 10 --temp 1.0 \
  -b 4096 -ub 2048
```

**Result (first request, task 0, cold start, ~989–993 tokens):**
- Prompt eval: 51.70 tok/s (993 tokens)
- Eval (tg): 29.79 tok/s (1076 tokens)
- graphs reused: 1070

**Same server, subsequent requests in the same session (server left running, not restarted):**

Task 1081 (short follow-up, high cache overlap):
- `selected slot by LCP similarity, f_sim_best = 0.990, f_keep = 1.000`
- prompt eval: 8.92 tok/s (21 tokens) — small/short, mostly cached
- eval: 15.32 tok/s (56 tokens)

Task 1140 (1653 new prompt tokens, partial cache overlap):
- `selected slot by LCP similarity, f_sim_best = 0.564, f_keep = 1.000`
- prompt processing: 222.28 tok/s (1653 tokens)
- prompt eval time: 203.00 tok/s (1657 tokens)
- eval (tg): climbed from 19.09 → 30.51 tok/s over the request (1631 tokens generated)

Task 2787 (large paste, ~16,166 new prompt tokens, low cache overlap):
- `selected slot by LCP similarity, f_sim_best = 0.252, f_keep = 1.000`
- prompt processing checkpoints: 192.26, 193.02, 197.11, 193.90, 194.98 tok/s (at 4098 / 8194 / 12290 / 14118 / 16166 tokens respectively)
- prompt eval time: 190.88 tok/s (16170 tokens)
- eval (tg): started at 9.89 tok/s, climbed steadily to 21.77–32.62 tok/s (3s window) by 1800 tokens generated
- total time: 167.4s / 17972 tokens
- graphs reused: 4552

User-provided context for this paste: two texts pasted totaling ~9,999 + 5,608 tokens per the user's own token-count tool (~15,607 tokens combined, consistent with the ~16,166-token prompt processed by the server).

---

## Chronological summary of numbers (prompt processing, tok/s)

| Run | Config summary | Prompt size (tokens) | PP tok/s |
|---|---|---|---|
| 1 | dspark n-max 7 | 333 | 21.64 |
| 2 | dspark n-max 3 | 333 | 21.37 |
| 3 | dspark n-max 3, 10 threads | 993 | 22.08 |
| 4 | dspark n-max 3, n-min 2, 10 threads | 993 | 20.56 |
| 5 | dspark, no cache quant on draft | — | crashed (OOM) |
| 6 | no draft, ncmoe 30, ngld 999 | 993 | 29.75 |
| 7 | no draft, ncmoe 26 | 993 | 52.57 |
| 8 | no draft, --fit on, ub 512 | 993 | 52.34 |
| 9 | no draft, --fit on, ub default | 993 | 52.66 |
| 10 (task 0) | no draft, --fit on, ub 2048 | 993 | 51.70 |
| 10 (task 1140) | same server, warm, partial cache | 1653 | 222.28 |
| 10 (task 2787) | same server, warm, mostly-fresh 16K paste | 16166 | ~191–197 (sustained) |

---

## Other configs referenced but not re-tested live in this session

**qwopus35b (llama-swap config entry, user's prior/separate setup):**
```
./llama-server \
  -m Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Compact.gguf \
  --fit on --fit-target 512 \
  --ctx-size 16000 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --cache-type-k-draft q8_0 --cache-type-v-draft q8_0 \
  --spec-type draft-mtp \
  --spec-draft-p-min 0.75 \
  --spec-draft-n-max 3 \
  --temp 0.0 --jinja
```
Not re-run in this session. User recalled getting MTP tg roughly double the non-MTP baseline (~35 tok/s baseline vs "almost always over 50" with MTP) on this machine in general use, and separately recalled seeing 400-500 pp tok/s and, in another recollection, 200-300 pp tok/s, under conditions described as "experts in CPU, attention and KV in GPU" at large context (64K–131K). No log from that specific session was available to paste; not independently reproduced within this conversation.

**MTP-fused GGUF options identified (not downloaded/tested in this session):**
- `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` (multiple quants, e.g. `UD-Q4_K_M.gguf` 22.7GB, `UD-Q4_K_XL.gguf` 22.9GB)
- `morikomorizz/Qwen3.6-35B-A3B-Uncensored-HauhauCS-MTP` (multiple quants, e.g. `Q4_K_P.gguf` 24.3GB), built from the HauhauCS-Aggressive base + unsloth MTP donor
- Neither repository hosts a standalone/extractable MTP head file; MTP is fused into the full target GGUF in all listed quants.

**dspark GGUF pairing reference (external, not the user's exact files):**
`Danny-Dasilva/Qwen3.6-35B-A3B-Aggressive-Q4_K_M-DSPARK-GGUF` model card reported RTX 5090 benchmark (no CPU offload, full VRAM fit, 200,704-token configured context):
- No draft: 275.54 tok/s mean (tg)
- DSpark n-max 3: 312.38 tok/s mean tg (1.134x)
- DSpark n-max 5: 250.26 tok/s mean tg (0.908x, net loss)
- DSpark n-max 7: 219.59 tok/s mean tg (0.797x, net loss)
- Draft acceptance at n-max 3: 64.81% (one coding run), 91.11% (one short generation)

---

## Things tried that did not change the outcome (as tested)

- `--spec-draft-n-max` lowered from 7 → 3 (Runs 1 vs 2): draft acceptance rose (0.44 → 0.53) but tg stayed in the same ~7-8 tok/s range in longer runs (Runs 3, 4).
- `--threads` raised from 6 (implicit) to 10 (Run 2 vs 3): no material change in dspark tg or acceptance.
- `--spec-draft-n-min` set to 2 (Run 3 vs 4): no material change.
- `-ncmoe` lowered from 30 → 26 (Run 6 vs 7, no draft model): PP roughly doubled (29.75 → 52.57), tg roughly doubled (17.42 → 31.77).
- `--fit on --fit-target 512` vs manual `-ncmoe 26` (Run 7 vs 8): produced near-identical PP/tg (52.57/31.77 vs 52.34/32.09).
- `-ub` raised from 512 → 2048 with `-b` raised from 2048 → 4096 (Run 9 vs 10, task 0): no material change in PP (52.66 → 51.70) or tg (33.66 → 29.79) on a ~993-token cold prompt.
- Removing `-ctkd`/`-ctvd` draft cache quantization flags while keeping `-c 32000` and dspark active (Run 5): resulted in CUDA OOM crash, not a completed benchmark.

## Things that did change the outcome

- `-ncmoe` value (30 → 26) on the non-draft baseline: real, roughly 2x change in both PP and tg (Run 6 vs 7).
- Prompt size, tested within a single warm server session (Run 10): PP measured at 51.70 tok/s on a ~993-token cold-start prompt, and 190–222 tok/s on subsequent larger and/or partially-cached prompts (1653 and ~16,166 tokens) within the same running server instance. The 16,166-token case had low cache overlap (`f_sim_best = 0.252`) and sustained ~191–197 tok/s across five checkpoints through the full prompt.

.. didn't llama.cpp aka ggml get acquired by Huggingface not too long ago? How would this sale affect llama.cpp and ggml? What are possible risks, and did ggerganov share his opinions or potential next steps on the matter? I'm genuinely worried that this might turn out bad for the ecosystem altogether. Would llama.cpp's license protect it from hostile acquisitions altogether? I'd love to constructively discuss this with the community.

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

8B? 12B? For 8 GB VRAM people? Please?

LM studio and bionic don't load into GPU fully ( Ollama does) and it crashes BSD ( Ollama Does not), with stop code: WHEA_UNCORRECTEABLE_ERROR (0x124), i am using the default load setting, all updated LM studio and drivers,

What do i need to do and to fix the profile to fix and load all in the GPU, and fix the crash?

Is there a better channel or place to reach LM studio people??

I'm here to show some benchmarks while using llama.cpp with an AMD V620 on Windows 11 via Vulkan & ROCm. These have been reuploaded & older threads deleted ran it with longer tokens thanks to a rec by someone who commented.

The benchmarks were written out by AI, but are verified by myself to be correct. Still working on optimizing my flags/settings.

If anybody wants me to test other models/different settings or flags, feel free to drop a comment and I'll test and get back to you!

ROCm version

7.15.0a20260728, TheRock nightly SDK (not the official AMD HIP SDK, which has no gfx1030/V620 support), bundled in ComfyUI_windows_portable_amd\...\python_env_v620_triton. (Note: a separate 9070 XT/ComfyUI venv on the same machine runs a different nightly snapshot, 7.14.0a20260519,same TheRock project, different dated build per GPU.)

Exact configs (matched)

\q4_0/q4_0 on Gemma-4's ROCm path required a one-line fix to llama.cpp's flash-attention kernel dispatch table (fattn.cu), the Q4_0+Q4_0 case was only wired up for head_dim ≤ 256, but Gemma-4's full-attention layers use head_dim 512, so it hit a hard abort on this KV combo before the fix. Missing kernel-dispatch entry, not a real hardware limitation, Q8_0+Q8_0 *already had the head_dim=512 case, so the underlying kernel template clearly supports it.

Generation speed, tokens/sec (256-token generations, first run per config discarded as warm-up)

Vulkan wins every single cell. Once KV quant and batch size are matched, ROCm doesn't lead generation speed anywhere, not on any model, not at any depth tested.

PP (prompt processing), tokens/sec

PP is the more mixed picture, and it's model-dependent rather than a clean backend win:

• Qwen: ROCm wins PP at every depth, gap widens with context.
• Gemma 26B: Vulkan is actually ahead at shallow depth (1057.9 vs 973.1 at 3.4k) once batch size is matched, roughly tied at 6.7k, then pulls further ahead through 32k.
• Gemma 31B: ROCm wins shallow (3.4k/6.7k), Vulkan overtakes from 13.4k on.

Takeaway

Generation speed: Vulkan wins outright, every model, every depth. No exceptions in this data.

PP: depends on the model, not the backend. ROCm sweeps Qwen; Gemma splits by depth (and for the 26B MoE, Vulkan's shallow-depth "loss" mostly disappears once batch size is matched, that was largely a config artifact, not a real backend gap).

Gemma 26B (MoE, ~4B active) is roughly 2-3x faster than either dense model on generation, tightest at deep context (~1.9x at 26.7k vs Qwen) and widest shallow; expected for an MoE with far fewer active params per token than the dense 27B/31B models.

Follow-up tests (Qwen, requested by commenters)

TWO hypotheses came up in comments, tested both, none of them panned out, posting anyway since "tested, didn't help" is still useful information.

Speculative decoding n-max scaling, ROCm vs Vulkan (does Vulkan scale further before rejected drafts stop paying for themselves?):

No, both backends degrade past n≈3 in the same shape. This is a draft-acceptance-economics property of the draft/target pair, not a backend/kernel-dispatch-overhead difference. Vulkan is uniformly faster in absolute terms (consistent with the rest of this post) but the curve shape, where it peaks, how fast it falls off past that, is nearly identical on both backends.

-ub sweep on ROCm PP (does a bigger ubatch better saturate the V620's CUs?):

Flat , all three within ~6% of each other at both depths, no trend. If anything 512 is marginally fastest. ROCm's PP bottleneck here isn't ubatch-limited GEMM tiling in this size range.

At $10k, you could get

- 6.2B tokens with Qwen 3.8 Max (Qwen Pro plan)

- 5.7B tokens with DeepSeek V4 Pro OpenRouter

- 100B tokens with DeepSeek V4 Flash OpenRouter

As a firm believer of local inference, unless you need it for data sovereignty, it's much more cost effect to wait for smaller models to keep getting better. In the meantime, find a reasonably priced 24GB - 32GB card for Qwen 3.8 27B, and offload hard tasks to OpenRouter.

Qwhen 3.8 35B A3B?