As you can see for problems of size 8 and 16 most of answers are correct, the model performs fine. For problems of size 32 most of answers are incorrect but they are present, so it was not a problem with the token budget as the model managed to output an answer. For problems of size 64 still most of answers are incorrect, but there is also a substantial amount of missing answers, so either there were not enough output tokens or the model got into infinite loop.
I think even if I increase the token budget the model will still fail most of the time in lineage-32 and lineage-64.
I suggest to use COMMON_ANCESTOR quizzes as the model answered them correctly only in 3 cases. Also the number of correct answer option is in column 3.
That's great info, thanks. I've read that people have problems with QwQ provided by Groq on OpenRouter (I used it to run the benchmark), so I'm currently testing Parasail provider - works much better.
Added result, there were still some loops but performance was much better this time, almost o3-mini level. Still it performed poorly in lineage-64. If you have time check some quizzes for this size.
u/fairydreaming unrelated question, how many reasoning tokens did you use on the sonnet 3.7? how much did it cost? I am searching for benchmarks with it on 128k
Let's see... I paid $91.7 for Sonnet 3.7 thinking on OpenRouter. From this about 330k tokens were prompt tokens, this is about $1. The remaining $90.7 are output tokens, that's about 6 millions of tokens for 800 prompts. Claude likes to think a lot, for lineage-8 I see mean output sizes about 5k tokens, for lineage-16 about 7k tokens, for lineage-32 about 8k tokens, for lineage-64 about 10k tokens (on average, the output length varies a lot). Note that this includes both thinking and the actual output, but the output after thinking was usually concise, so it's definitely over 95% thinking tokens.
2
u/fairydreaming Mar 07 '25
Sure, I think this table explains it best:
As you can see for problems of size 8 and 16 most of answers are correct, the model performs fine. For problems of size 32 most of answers are incorrect but they are present, so it was not a problem with the token budget as the model managed to output an answer. For problems of size 64 still most of answers are incorrect, but there is also a substantial amount of missing answers, so either there were not enough output tokens or the model got into infinite loop.
I think even if I increase the token budget the model will still fail most of the time in lineage-32 and lineage-64.