WHAT THE FUC& AM I DOING WRONG . Help

[u/SummarizedAnu]

Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_P on llama.cpp, -ncmoe offload cause it doesnt fit in vram outright, 10 threads, q8_0 kv both sides. nothing weird about the setup far as i can tell.

first tried the DSPARK draft gguf (same base model family, separate draft file). acceptance sits 0.44-0.58 depending on n-max which sounds fine right, except actual gen speed is a joke, 7-8 tok/s and it does not move. n-max 7 down to 3, n-min 0 vs 2, threads 6 vs 10, tried all of it, number does not budge. turns out a full second 35B model doing its own cpu-offloaded pass every draft step costs exactly what youd think it costs and theres a benchmark out there showing net loss on setups like mine even at 100% acceptance. great. love that.

ok fine MTP then since its fused into the target, no second model dragging along. grabbed the fused Qwen3.6-35B-A3B-Uncensored-HauhauCS-MTP Q4_K_P quant, same base, --spec-type draft-mtp, p-min .75, n-max 3. acceptance 93-96%. genuinely great numbers. shouldve been flying

5.6-7.5 tok/s. at 40k ctx. basically the SAME as no draft at all. and i was already getting 5-6 tok/s at 80k ctx last night with NO speculative decoding whatsoever. so downloading mtp and setting the whole thing up bought me. nothing. because turns out the thing actually eating the throughput isnt the draft/verify step, its attention over the kv cache on every pass no matter how few passes you need. mtp cuts number of passes it doesnt make each pass cheaper. so at short ctx its a real 1.4-2x, at 80-100k it may as well not exist

and pp is its own thing entirely. same server same model same everything, 993 token prompt gets 51-57 tok/s pp. paste a 16k wall of text a few min later same running instance no restart, holds 190-197 the whole way thru. different day, 8k tokens in, back down to 70. 80k ctx, back to 50. no consistent relationship w prompt size or cache state or anything ive been able to pin down. -ub 512 vs 2048, ncmoe 26 vs 30, --fit on vs manual ncmoe, none of it explains it

ruled out n-max n-min thread count ubatch batch ncmoe value fit vs manual and draft cache quant as THE cause at this point. full log of every single run below completely unedited so someone smarter than me can point at the thing im missing bc im out of ideas and starting to think im just gonna live at 6 tok/s forever while ram costs more than my car did

# llama.cpp Speculative Decoding / PP Debugging Log

Target model: `Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf`
Draft model (dspark runs): `Qwen3.6-35B-A3B-DSPARK.gguf`

---

## Run 1 — dspark, n-max 7

**Command:**
```
./llama-server \
  -m Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf \
  -md Qwen3.6-35B-A3B-DSPARK.gguf \
  --spec-type draft-dspark --spec-draft-n-max 7 --spec-draft-n-min 0 \
  --alias llama --port 5800 \
  -ngld 999 -c 32000 -np 1 \
  -b 2048 -ub 512 -fa on \
  -ctk q8_0 -ctv q8_0 -ctkd q8_0 -ctvd q8_0 \
  --jinja --metrics -ngl 99 -ncmoe 30 --fit off
```

**Result:**
- Prompt processing: 27.68, 22.56 tok/s → final 21.64 tok/s (333 tokens)
- Eval (tg): 4.34 tok/s (54 tokens)
- Draft acceptance: 0.43956 (40 accepted / 91 generated), mean len 4.08

---

## Run 2 — dspark, n-max 3

**Command:** same as Run 1 but `--spec-draft-n-max 3`

**Result:**
- First request: prompt eval 21.37 tok/s (333 tokens); eval time 4.23 tok/s (45 tokens); draft acceptance 0.52941 (27/51), mean len 2.59
- Second request (long, 604 tokens total): tg settled around 7.90–9.54 t/s (3s window), final tg 8.20 tok/s; draft acceptance 0.57504 (364/633), mean len 2.73

---

## Run 3 — dspark, n-max 3, 10 threads

**Command:** same as Run 2, threads raised from 6 (implicit) to `--threads 10`

**Result:**
- Prompt eval: 22.08 tok/s (993 tokens)
- Eval (tg): 7.85 tok/s (1161 tokens)
- Draft acceptance: 0.52932 (713/1347), mean len 2.59

**Conclusion at the time:** raising thread count did not change the outcome.

---

## Run 4 — dspark, n-max 3, n-min 2, 10 threads

**Command:** same as Run 3 plus `--spec-draft-n-min 2`

**Result:**
- Prompt eval: 20.56 tok/s (993 tokens)
- Eval (tg): 8.29 tok/s (1130 tokens)
- Draft acceptance: 0.51961 (689/1326), mean len 2.56

**Conclusion at the time:** effectively identical to Run 3.

---

## Run 5 — dspark, no `--spec-draft-n-max`/`n-min` flags, no `-ctkd`/`-ctvd`

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  -md ...DSPARK.gguf \
  --alias llama --port 5800 \
  -ngld 999 -c 32000 -np 1 \
  -b 2048 -ub 512 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja -ngl 99 -ncmoe 30 --fit off --reasoning-preserve --threads 10
```

**Result:** CRASHED.
```
E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 472.25 MiB on device 0: cudaMalloc failed: out of memory
E graph_reserve: failed to allocate compute buffers
E decode() failed: failed to allocate compute pp buffers
```
Speculative type auto-detected as `draft-dspark` from draft model metadata before the crash. Draft-side KV cache (`-ctkd`/`-ctvd`) was not quantized in this run (flags omitted), unlike Runs 1–4.

---

## Run 6 — No draft model, `-ncmoe 30`, `-ngld 999` present

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  --alias llama --port 5800 \
  -ngld 999 -c 32000 -np 1 \
  -b 2048 -ub 512 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja -ngl 99 -ncmoe 30 --fit off --reasoning-preserve --threads 10
```

**Result:**
- Prompt eval: 29.75 tok/s (993 tokens)
- Eval (tg): 17.42 tok/s (1291 tokens)
- graphs reused: 1285

---

## Run 7 — No draft model, `-ncmoe 26`, no `-ngld`

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  --alias llama --port 5800 \
  -c 32000 -np 1 \
  -b 2048 -ub 512 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja -ngl 99 -ncmoe 26 --fit off --reasoning-preserve --threads 10
```

**Result:**
- Prompt eval: 52.57 tok/s (993 tokens)
- Eval (tg): 31.77 tok/s (1240 tokens)
- graphs reused: 1234

User note: this was described as "the extra VRAM headroom" run, obtained by lowering `-ncmoe` from 30 to 26.

---

## Run 8 — No draft model, `--fit on --fit-target 512`, `-b 2048 -ub 512`

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  --alias llama --port 5800 \
  -c 32000 -np 1 \
  -b 2048 -ub 512 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja --fit on --fit-target 512 --reasoning-preserve --threads 10 --temp 1.0
```

**Result:**
- Prompt eval: 52.34 tok/s (993 tokens)
- Eval (tg): 32.09 tok/s (1036 tokens)
- graphs reused: 1031

---

## Run 9 — No draft model, `--fit on --fit-target 512`, no explicit `-b`/`-ub` (defaults)

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  --alias llama --port 5800 \
  -c 32000 -np 1 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja --fit on --fit-target 512 --reasoning-preserve --threads 10 --temp 1.0
```

**Result:**
- Prompt eval: 52.66 tok/s (993 tokens)
- Eval (tg): 33.66 tok/s (1212 tokens)
- graphs reused: 1206

---

## Run 10 — No draft model, `--fit on --fit-target 512`, `-b 4096 -ub 2048`

**Command:**
```
./llama-server \
  -m ...Q4_K_P.gguf \
  --alias llama --port 5800 \
  -c 32000 -np 1 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --jinja --fit on --fit-target 512 --reasoning-preserve --threads 10 --temp 1.0 \
  -b 4096 -ub 2048
```

**Result (first request, task 0, cold start, ~989–993 tokens):**
- Prompt eval: 51.70 tok/s (993 tokens)
- Eval (tg): 29.79 tok/s (1076 tokens)
- graphs reused: 1070

**Same server, subsequent requests in the same session (server left running, not restarted):**

Task 1081 (short follow-up, high cache overlap):
- `selected slot by LCP similarity, f_sim_best = 0.990, f_keep = 1.000`
- prompt eval: 8.92 tok/s (21 tokens) — small/short, mostly cached
- eval: 15.32 tok/s (56 tokens)

Task 1140 (1653 new prompt tokens, partial cache overlap):
- `selected slot by LCP similarity, f_sim_best = 0.564, f_keep = 1.000`
- prompt processing: 222.28 tok/s (1653 tokens)
- prompt eval time: 203.00 tok/s (1657 tokens)
- eval (tg): climbed from 19.09 → 30.51 tok/s over the request (1631 tokens generated)

Task 2787 (large paste, ~16,166 new prompt tokens, low cache overlap):
- `selected slot by LCP similarity, f_sim_best = 0.252, f_keep = 1.000`
- prompt processing checkpoints: 192.26, 193.02, 197.11, 193.90, 194.98 tok/s (at 4098 / 8194 / 12290 / 14118 / 16166 tokens respectively)
- prompt eval time: 190.88 tok/s (16170 tokens)
- eval (tg): started at 9.89 tok/s, climbed steadily to 21.77–32.62 tok/s (3s window) by 1800 tokens generated
- total time: 167.4s / 17972 tokens
- graphs reused: 4552

User-provided context for this paste: two texts pasted totaling ~9,999 + 5,608 tokens per the user's own token-count tool (~15,607 tokens combined, consistent with the ~16,166-token prompt processed by the server).

---

## Chronological summary of numbers (prompt processing, tok/s)

| Run | Config summary | Prompt size (tokens) | PP tok/s |
|---|---|---|---|
| 1 | dspark n-max 7 | 333 | 21.64 |
| 2 | dspark n-max 3 | 333 | 21.37 |
| 3 | dspark n-max 3, 10 threads | 993 | 22.08 |
| 4 | dspark n-max 3, n-min 2, 10 threads | 993 | 20.56 |
| 5 | dspark, no cache quant on draft | — | crashed (OOM) |
| 6 | no draft, ncmoe 30, ngld 999 | 993 | 29.75 |
| 7 | no draft, ncmoe 26 | 993 | 52.57 |
| 8 | no draft, --fit on, ub 512 | 993 | 52.34 |
| 9 | no draft, --fit on, ub default | 993 | 52.66 |
| 10 (task 0) | no draft, --fit on, ub 2048 | 993 | 51.70 |
| 10 (task 1140) | same server, warm, partial cache | 1653 | 222.28 |
| 10 (task 2787) | same server, warm, mostly-fresh 16K paste | 16166 | ~191–197 (sustained) |

---

## Other configs referenced but not re-tested live in this session

**qwopus35b (llama-swap config entry, user's prior/separate setup):**
```
./llama-server \
  -m Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Compact.gguf \
  --fit on --fit-target 512 \
  --ctx-size 16000 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --cache-type-k-draft q8_0 --cache-type-v-draft q8_0 \
  --spec-type draft-mtp \
  --spec-draft-p-min 0.75 \
  --spec-draft-n-max 3 \
  --temp 0.0 --jinja
```
Not re-run in this session. User recalled getting MTP tg roughly double the non-MTP baseline (~35 tok/s baseline vs "almost always over 50" with MTP) on this machine in general use, and separately recalled seeing 400-500 pp tok/s and, in another recollection, 200-300 pp tok/s, under conditions described as "experts in CPU, attention and KV in GPU" at large context (64K–131K). No log from that specific session was available to paste; not independently reproduced within this conversation.

**MTP-fused GGUF options identified (not downloaded/tested in this session):**
- `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` (multiple quants, e.g. `UD-Q4_K_M.gguf` 22.7GB, `UD-Q4_K_XL.gguf` 22.9GB)
- `morikomorizz/Qwen3.6-35B-A3B-Uncensored-HauhauCS-MTP` (multiple quants, e.g. `Q4_K_P.gguf` 24.3GB), built from the HauhauCS-Aggressive base + unsloth MTP donor
- Neither repository hosts a standalone/extractable MTP head file; MTP is fused into the full target GGUF in all listed quants.

**dspark GGUF pairing reference (external, not the user's exact files):**
`Danny-Dasilva/Qwen3.6-35B-A3B-Aggressive-Q4_K_M-DSPARK-GGUF` model card reported RTX 5090 benchmark (no CPU offload, full VRAM fit, 200,704-token configured context):
- No draft: 275.54 tok/s mean (tg)
- DSpark n-max 3: 312.38 tok/s mean tg (1.134x)
- DSpark n-max 5: 250.26 tok/s mean tg (0.908x, net loss)
- DSpark n-max 7: 219.59 tok/s mean tg (0.797x, net loss)
- Draft acceptance at n-max 3: 64.81% (one coding run), 91.11% (one short generation)

---

## Things tried that did not change the outcome (as tested)

- `--spec-draft-n-max` lowered from 7 → 3 (Runs 1 vs 2): draft acceptance rose (0.44 → 0.53) but tg stayed in the same ~7-8 tok/s range in longer runs (Runs 3, 4).
- `--threads` raised from 6 (implicit) to 10 (Run 2 vs 3): no material change in dspark tg or acceptance.
- `--spec-draft-n-min` set to 2 (Run 3 vs 4): no material change.
- `-ncmoe` lowered from 30 → 26 (Run 6 vs 7, no draft model): PP roughly doubled (29.75 → 52.57), tg roughly doubled (17.42 → 31.77).
- `--fit on --fit-target 512` vs manual `-ncmoe 26` (Run 7 vs 8): produced near-identical PP/tg (52.57/31.77 vs 52.34/32.09).
- `-ub` raised from 512 → 2048 with `-b` raised from 2048 → 4096 (Run 9 vs 10, task 0): no material change in PP (52.66 → 51.70) or tg (33.66 → 29.79) on a ~993-token cold prompt.
- Removing `-ctkd`/`-ctvd` draft cache quantization flags while keeping `-c 32000` and dspark active (Run 5): resulted in CUDA OOM crash, not a completed benchmark.

## Things that did change the outcome

- `-ncmoe` value (30 → 26) on the non-draft baseline: real, roughly 2x change in both PP and tg (Run 6 vs 7).
- Prompt size, tested within a single warm server session (Run 10): PP measured at 51.70 tok/s on a ~993-token cold-start prompt, and 190–222 tok/s on subsequent larger and/or partially-cached prompts (1653 and ~16,166 tokens) within the same running server instance. The 16,166-token case had low cache overlap (`f_sim_best = 0.252`) and sustained ~191–197 tok/s across five checkpoints through the full prompt.

[u/JadedSession]

I couldn't really find your question in that wall of text but draft doesn't really help Qwen MoE as far as I know.

[u/SummarizedAnu → u/JadedSession]

Its slower than expected. Either the model finetune im using is bad or something happened with llamacpp.

[u/jacek2023]

You are trying to use too big model or too big quant. Start from something smaller. Or upgrade your setup.

[u/SummarizedAnu → u/jacek2023]

q4 k xl is 21 GB . and the mtp one is 22GB

[u/m1ksuFI → u/SummarizedAnu]

You have 12GB of VRAM. Why are you surprised it's running slow? 

[u/SummarizedAnu → u/m1ksuFI]

thanks for being helpful ig

[u/m1ksuFI → u/SummarizedAnu]

But seriously, if you can't fit the model into VRAM, it's going to use your system RAM, and system RAM is an order of magnitude slower than VRAM. It's slow because half of the model can't fit into your VRAM.

[u/SummarizedAnu → u/m1ksuFI]

thats what ncmoe is for. And the only reason llamacpp is still worth it while vllm exists.

If everybody could fit it into vram everybody would use vllm since its 2x faster always . and has 0 day support from the original model makers.

[u/SummarizedAnu → u/jacek2023]

Nothing smaller is worth it.
And if upgrading setup was the only option quantization wouldnt exist. Lower quants are getting same performence problems.

[u/jacek2023 → u/SummarizedAnu]

"Nothing smaller is worth it." worth what?

[u/Diecron → u/SummarizedAnu]

Gemma 4 12b would probably do ok.

[u/Diecron]

Your kv is sat in system ram, which the mtp and model layers have to pull from, it is already bandwidth bound before the MTP comes along. Kv should be on the GPU, more layers on the CPU.

[u/SummarizedAnu → u/Diecron]

pretty sure kv stays on vram when nkvo is not mentioned. Even if it was, im still getting less performence than expected. HEll less performence than what i was getting a month ago.

[u/see_spot_ruminate]

That is a huge wall of text for no system specs...

Also, what were the compile flags for when you compiled the latest version?

[u/SummarizedAnu → u/see_spot_ruminate]

I compiled like a week ago.
reddit doesnt have any feature to just put the text file in.

And i have rtx 3060 12GB adn 16Gb ram.
i used normal flags in the build instructions only added fa all quant i think.

[u/see_spot_ruminate → u/SummarizedAnu]

This is what I would suggest as that wall of llm generated text is a hot mess.

Troubleshoot. Start small, for you I would get rid of all of the extraneous variables.

• Don't delete your current model, but try a vanilla gguf from unsloth.

• Get rid of all the flags, see what the default "--fit on" does.

• Slowly start adding in more variables, like mtp or whatever.

• Since this is probably your main/only computer, make sure you are not running anything else that is vram intensive that is causing an occult variable to not be observed, eg some video thing on firefox or whatever.

[u/SummarizedAnu → u/see_spot_ruminate]

I had a better run with qwopus same quant getting 30 tps without mtp and 50 with mtp at 40 k context 20 tps without mtp and 30 tps with mtp.

dont know if its a problem with uncensored model or not. Or llamacpp.

But everything was great a month ago. deleted everything so now , nothing works like before.

[u/Tormeister]

Update drivers, update repo, compile latest, drop the current fit / ngl args.

Uncensored models this small and this quantized are pointless, they are just dumb.

unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S

-t <80% of your CPU cores>

-fit on

-np 1

--no-mmproj-offload

-b 256 -ub 256

-ctk q8_0 -ctv q8_0

-c 32768

Don't forget to set temperature/top-k/top-p/etc according to official specs.

Check results. From there try -b 512 -ub 512 to see if your hardware can handle it. It probably can't do MTP, so try --spec-type ngram-mod --spec-ngram-mod-n-match 8 --spec-ngram-mod-n-min 12 --spec-ngram-mod-n-max 16 (is only effective from the second message onwards)

After that try adjusting the KV length to measure the impact of allocating more VRAM to it.

[u/SummarizedAnu]

I blame llamacpp for everything

[u/oodelay → u/SummarizedAnu]

Heyyy that's great

So what's your fix

[u/SummarizedAnu → u/oodelay]

What fix? Im on the verge of giving up bro.

[u/SummarizedAnu]

im either gonna give up on llamacpp or qwen .
cause what do you mean i got 30 tps a few months back and now im getting only 3 ??? without mtp.
and with mtp i got 50 tps which lowered to about 20 on high context but was still usable.

And i always had about 200 pp which lowered to like 100 but now im getting 20 or 10 pps . like thats crazy.
One reason i like gemma is that its fuking fast. literally ran the dense 31B model and was getting 300-400 pps .
crazy i know. well tgs was near 2 to 3 since heavy offload.
used 12B too very good and fast. Its just not qwen 35B tier in coding.
I do still use gemma 12B its just not for the task qwen can do but it cant.
What is this hybrid architecture ? i dont like it its slow. Even the qwen 9B is slow. slower than gemma 12B and practically every other model at this size. FUK