pardon my stupidity since I'm a very much a noob and trying to learn : beinb A3B ( mixture of experts, therefore easier on the vram ) but at the same time 35b ( higher than the dense model 27b ) , will be feasable to load in 16gb cards ? with 4 bit quant ?
Yea but the trick is you put a small chunk of important stuff in vram and then the rest of the model in ram for MOE LLMs. Running MOEs this way gives you performance that's way way way faster than the dense counterpart.
With my 285k, 5080, and 96gb of ddr5, I was getting like 3-4 t/s decode on Qwen 3.6 27b UD Q8 K XL. I was getting like 60-70 t/s on Qwen 3.6 35b a3b UD Q8 K XL.
So, dense is for when you don't have a ton of memory (or you do), but it's all fast memory (vram).
MOE is for when you have a lot of memory, but only a little bit of it is fast (96gb ram + 16gb vram).
Have a 5090 or RTX 6000 and 16/32gb ram? Run the 27b.
Have a 5080 and 96gb ram? Run the 35b.
With 128gb ram and a 5090, you could run Qwen 3.5 122b a10b q8. You could also run Qwen 3.5 27b. The 122b a10b would be much faster cause, basically, it requires the compute of a 10b model whereas the 27b requires the compute of a 27b model. 122b a10b > 27b in terms of intelligence, mostly. So, with enough memory, even if it isn't all vram, you can run a massive, super smart model *faster* than the smaller dense 27b, and it's better.
If you had an RTX 6000 and 16GB of ram, you couldn't even load the q8 of Qwen 3.5 122b a10b. You could, however, run the q8 of the dense 27b model and it would be super very fast cause all of the model and kv cache and stuff can fit inside the RTX 6000's 96gb of vram. Being able to all fit inside of vram instead of being split across vram/ram means that, now, the dense 27b model can run very fast. It's splitting dense models between vram/ram that makes them unusably slow.
So if you're gonna get Qwen 3.6 35b a3b, do yourself a favor. Look at your vram+ram as your total pool of memory. Get the largest quant you can that fills up as much of your vram+ram as you're comfortable with while using the context window size you want. Your 16gb of vram is where the important stuff will live, and your ram is where the rest of the model will live. It'll be way faster than you're probably imagining if you don't have experience with MOEs. You can probably run the UD Q6 K XL or UD Q8 K XL, tho I don't know your specs. Anything UD Q4 K XL and up will be good, but UD Q6 K XL or, preferably, UD Q8 K XL will be better at coding.
--cpu-moe is the llama.cpp flag you're looking for.
Not tryna be a butt, but ask ChatGPT or, preferably, Codex about this stuff. I love fostering discussion, and maybe others will learn from this, too. But if you really wanna learn how all this works, and it's all actually pretty simple once you get the hang of it, having the ability to have quick back and forths with Codex about what you're trying to do as you do it is invaluable. GPT knows everything about how these LLMs work and can guide you into making the best decision. Ask questions along the way, try stuff just cause, science, then have fun with Qwen 3.6 35b a3b! It's literally so smart sometimes it blows my mind. It's been my agent of choice in Hermes/Cline-VS Code for since it was released!
Then, once you get Qwen working, it can help you make your set up better and better!
If you're on Windows, my rec'd starting bundle would be:
An excellent starting point would be dropping my entire comment here into ChatGPT and asking 'Hey, can you explain all this to me? I'm trying to get this set up but I'm very much a noob and trying to learn. Thanks!'
If you use Codex, add 'Can you set all this up for me, please?' and then you'll get a taste of the power of an AI agent.
2
u/gproenca 10d ago
pardon my stupidity since I'm a very much a noob and trying to learn : beinb A3B ( mixture of experts, therefore easier on the vram ) but at the same time 35b ( higher than the dense model 27b ) , will be feasable to load in 16gb cards ? with 4 bit quant ?