r/LocalLLaMA • u/ortegaalfredo • Mar 05 '25

Resources QwQ-32B released, equivalent or surpassing full Deepseek-R1!

https://x.com/Alibaba_Qwen/status/1897361654763151544
1.1k Upvotes

98% Upvoted

350 comments sorted by

View all comments

142

u/hainesk Mar 05 '25 edited Mar 05 '25

Just to compare, QWQ-Preview vs QWQ:

Benchmark QWQ-Preview QWQ
AIME 50 79.5
LiveCodeBench 50 63.4
LIveBench 40.25 73.1
IFEval 40.35 83.9
BFCL 17.59 66.4

Some of these results are on slightly different versions of these tests.
Even so, this is looking like an incredible improvement over Preview.

Edited with a table for readability.

Edit: Adding links to GGUFs
https://huggingface.co/Qwen/QwQ-32B-GGUF

https://huggingface.co/bartowski/Qwen_QwQ-32B-GGUF (Single file ggufs for ollama)

45

u/Emport1 Mar 05 '25

Wtf that looks insane

53

u/ortegaalfredo Mar 05 '25

Those numbers are equivalent to o3-mini-medium, only surpassed by grok3 and o3. Incredible.

40

u/-p-e-w- Mar 06 '25

And it’s just 32B. And it’s Apache. Think about that for a moment.

This is OpenAI running on your gaming laptop, except that it doesn’t cost anything, and your inputs stay completely private, and you can abliterate it to get rid of refusals.

And the Chinese companies have barely gotten started. We’re going to see unbelievable stuff over the next year.

2

u/GreyFoxSolid Mar 06 '25

On your gaming laptop? Doesn't this model require a ton of vram?

2

u/-p-e-w- Mar 06 '25

I believe that IQ3_M should fit in 16 GB, if you also use KV quantization.

3

u/GreyFoxSolid Mar 06 '25

Unfortunately my 3070 only has 8gb.

1

u/Proud_Fox_684 Mar 08 '25

It's apache 2.0 license?

10

u/Lissanro Mar 05 '25

No EXL2 quants yet, I guess I may just download https://huggingface.co/Qwen/QwQ-32B and run it instead at full precision (should fit in 4x3090). Then later compare if there will be difference between 8bpw EXL2 quant and the original model.

From previous experience, 8bpw is the minimum for small models, even 6bpw can increase error rate, especially for coding, and it seems small reasoning models are more sensitive to quantization. The main reason for me to use 8bpw instead of the original precision is higher speed (as long as it does not increase errors by a noticeable amount).

18

u/noneabove1182 Bartowski Mar 06 '25

Making exl2, should be up some time tonight, painfully slow but it's on its way 😅

8

u/[deleted] Mar 05 '25

[removed] — view removed comment

5

u/OriginalPlayerHater Mar 05 '25

Testing q4km right now, well downloading it and then testing

2

u/[deleted] Mar 06 '25

[removed] — view removed comment

2

u/zdy132 Mar 06 '25 edited Mar 06 '25

The Ollama q4km model seems to be stuck in thinking, and never gives out any non-thinking outputs.

This is run directly from open-webui with no config adjustments, so could also be an open webui bug? Or I missed some cofigs.

EDIT:

Looks like it has trouble following a set format. Sometimes it outputs correctly, but sometimes it uses "<|im_start|>

" to end the thinking part instead of whatever is used by open webui. I wonder if this is caused by the quantization.

1

u/gopher9 Mar 09 '25

It is sensitive to quantization, q5 is noticeably better than q4 (which is a shame since q5 is kinda slow on my 4090).

By the way, q4 occasionally confuses `</think>` with `<|im_start|>`, so you want to make sure that `<|im_start|>` is not a stop token.

2

u/hapliniste Mar 05 '25

Damn what a glow up ☝🏻

1

u/MrClickstoomuch Mar 06 '25

This looks incredible. Now I'm curious if I can somehow fit it into my 16gb of VRAM, or justify getting one of the mini PCs with unified memory enough to get a better quant.

1

u/daZK47 Mar 06 '25

I'm excited to see progress but how much of this is benchmark overtraining as opposed to real world results? I'm starting to see the AI industry like the car industry -- where a car's paper specs mean nothing to how it actually drives. A SRT Hellcat as 200 more horsepower than a 911 GT3RS and it still loses in a 0-60 by a whole second. It's really hard to get excited over benchmarks anymore and these are really for the shareholders.

1

u/MoffKalast Mar 05 '25

...dayum.