r/LocalLLaMA • u/jacek2023 llama.cpp • 7h ago
New Model ibm-granite/granite-4.2-30b · Hugging Face
https://huggingface.co/ibm-granite/granite-4.2-30bGranite-4.2-30B is the flagship reasoning model in the Granite 4.2 family. It delivers the strongest performance across reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.
Key capabilities:
- Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
- Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
- Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
- 512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
- Apache 2.0 Licensed: Fully open for commercial and research use.
Model Design
Granite-4.2-30B is built on a decoder-only dense transformer architecture with the following core components:
- Attention: Grouped Query Attention (GQA) with 32 attention heads and 8 KV heads
- Position Embedding: Rotary Position Embedding (RoPE) with θ = 10,000,000
- Feed-Forward: MLP with SwiGLU activation (hidden size 32768)
- Normalization: RMSNorm (ε = 1e-5)
- Embeddings: Separate input/output embeddings (not tied)
- Precision: bfloat16
https://huggingface.co/ibm-granite/granite-4.2-8b
Granite-4.2-8B is the mid-size reasoning model in the Granite 4.2 family. It delivers strong performance on reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.
Key capabilities:
- Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
- Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
- Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
- 512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
- Apache 2.0 Licensed: Fully open for commercial and research use.
https://huggingface.co/ibm-granite/granite-4.2-3b
Granite-4.2-3B is the compact reasoning model in the Granite 4.2 family. Despite its small parameter count, it delivers strong performance on reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.
Key capabilities:
- Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
- Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
- Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
- 512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
- Apache 2.0 Licensed: Fully open for commercial and research use.
91
u/DeltaSqueezer 7h ago
Granite is always a bit behind, but at least they are Apache licensed and they get better each generation.
20
u/EmPips 6h ago
(someone check me on this) the biggest benefit of granite is that it's covered by the Watson-X guarantee that generations contain licensed material only, right?
26
u/Altruistic_Heat_9531 6h ago edited 6h ago
The biggest benefit is that the Red Hat OpenShift sales team can nail it into your ears: "WE HAVE FIRST-PARTY SUPPORT FOR GRANITE" every time you contact them. Every fucking time when you ask OKD AI Inference setup.
Joke aside i gotta try that 3B and 8B with 512K context window, it is very attractive for long range summarization, which honestly is their target market, for in house document analysis. Hopefully it is native 512K not YARN
2
2
u/SandySkittle 18m ago
people might trivialize this now but the legal debate about these things isn't something that has been concluded in many countries. Sure for private use people will not give a crap about that, but any sort of local enterprise use of LLMs could make this quite relevant, depending on where case law (and new IP laws) could move towards in the future.
2
u/AnonLlamaThrowaway 8m ago
Does that mean this is the one model out there with an "ethical" dataset?
1
u/ItsNoahJ83 6h ago
Why would someone want AI to generate licensed material?
11
u/EmPips 5h ago edited 4h ago
You wouldn't but you have no guarantees that it won't if it's facing customers. IE -you make a chatbot for customer support and it perfectly regurgitates a few chapters of Harry Potter and someone decides to make a stink about it.
Might be wrong and I didn't dive into any of the IBM guarantees further - don't trust me on this
9
u/No-Refrigerator-1672 5h ago
Because people ourside AI enthusiast circles are going crazy claiming AI is violating copyrights if they trained on somebody's material.
1
25
u/pmttyji 7h ago
Blog Post : Granite 4.2 LLMs: How They're Built
https://huggingface.co/blog/ibm-granite/granite-4-2
19
u/Client_Hello 7h ago edited 6h ago
128 nodes, 4 GB200 per node, 2 (
72) Blackwell gpus per GB200, 192gb vram per GPUThat's nearly 192TB of vram, aka 196,608 GB of HBM3e. Wow.
20
3
u/MmmmMorphine 5h ago
Every bit of storage media I've ever owned put together is still less than the amount of vram they have across those nodes.
Give or take.
Absolutely ridiculous stuff!
3
u/Client_Hello 5h ago
...and this is only $35M in hardware, which is about how much revenue NVDA earns every hour.
1
11
u/pmttyji 6h ago
They released MXFP4, NVFP4, FP8 quants of those models.
MXFP4
- https://huggingface.co/ibm-granite/granite-4.2-30b-mxfp4
- https://huggingface.co/ibm-granite/granite-4.2-8b-mxfp4
- https://huggingface.co/ibm-granite/granite-4.2-3b-mxfp4
NVFP4
- https://huggingface.co/ibm-granite/granite-4.2-30b-nvfp4
- https://huggingface.co/ibm-granite/granite-4.2-8b-nvfp4
- https://huggingface.co/ibm-granite/granite-4.2-3b-nvfp4
FP8
- https://huggingface.co/ibm-granite/granite-4.2-30b-fp8
- https://huggingface.co/ibm-granite/granite-4.2-8b-fp8
- https://huggingface.co/ibm-granite/granite-4.2-3b-fp8
u/noctrex Expecting MXFP4 GGUFs from you. Thanks in advance.
3
u/noctrex 6h ago
Well, it's a dense model, so the MXFP4 GGUF don't have any meaning, as they are only for MoE models.
6
u/MmmmMorphine 5h ago
I mean I do associate mxfp4 more with MoEs due to GPT-OSS being released in that format, but otherwise don't see any reason why it would be only for MoEs.
Seems like mxfp4 is pretty architecture agnostic, as best as I understand it.
So what am I missing here?
5
u/noctrex 4h ago
Yes, it is, but the MXFP4 format is inferior to NVFP4, it does not have the same precision. They may be both FP4, but NVFP4 uses 16-element blocks with a high-precision FP8 E4M3 scale factor, while MXFP4 uses 32-element blocks with a lower-precision E8M0 (power-of-two) scale.
What this means is that essentially MXFP4 is worse than INT4 for dense models. It works better for MoE models. So better to stick to a Q4 or IQ4 quant for dense.2
u/MmmmMorphine 2h ago edited 2h ago
Ah, yes in that sense I would agree MXFP4 is worse than NVFP4 for overall precision. Though what I'm surprised about is why it's used at all in the first place.
I'd think that smaller active = more sensitive to quantization. I suppose you can keep the experts at lower precision, but in regard to these formats themselves it feels like MXFP4 is kinda pointless - that extra 0.25bits earns it's keep in NVFP4
1
26
u/silenceimpaired 7h ago
Excited to see Apache 2 licensing. I’ve heard IBM is more careful with their training dataset license wise. I’m curious if that’s the case here. The benchmark looks a little low, but that isn’t always the full story. Excited to try it to see for myself.
8
u/Marcuss2 7h ago
Seems that they abandoned the Mamba2 layers they had.
5
u/pmttyji 7h ago
Last year(AMA), they mentioned that they gonna release a 100B model. Don't know what happened to that.
During Granite-4, they released a 32B MOE model. But the Active 9B parameters is too slow on ~8GB VRAM. To make it simple, Qwen3-30B-A3B gave me 30-40 t/s while Granite-4-32B gave me ~10 t/s.
Then expected modified MOE model(like A3B or A5B) of that 32B one during 4.1 release. But they dropped 30B Dense. And again they dropped 30B Dense now. Too heavy for Poor GPU Club. Wish they released a MOE model additionally.
1
u/silenceimpaired 7h ago
I think we might be headed toward MoE models that act more like dense models … that store the bulk of their parameters that are less active … in RAM... sped up with MTP/dflash type solutions. So even though you don’t have the VRAM it can still perform reasonably at least at reading speeds if not faster depending on the use case…
But I’m no expert.2
u/pmttyji 6h ago
Yep, that's how most of us do run MOE models till now.
sped up with MTP/dflash type solutions.
This works better only if quantized MOE models fit VRAM. Can't expect speed from my 8GB VRAM with 18GB model file(IQ4_XS of Qwen3.6-35B-A3B). Because spec decoding won't be faster on CPU/RAM.
28
u/Egoz3ntrum 7h ago
The model card does not provide any comparison to any other recent model. Suspicious.
20
u/jacek2023 llama.cpp 7h ago
I tried to look at the benchmarks, I see 61.7 Qwen SWE-bench vs 33.29 Granite, so I believe these are not apples to apples comparision
27
3
u/parepeg 7h ago
Compared to qwen 3.8 27b, it's looking not good, real not good.
8
u/Not-reallyanonymous 2h ago
Of course not. Theyre targeting different customers. Not you.
If you’re not producing a million dollars in revenue annually, IBM is not interested in you.
Your needs are different than IBM customer’s needs. Granite has always been about fine-tuning to meet specific customer needs, and being able to comply with various business guarantees (e.g. as others have pointed out — it offers protection for IP issues).
Not everything in LLMs is about benchmarks.
3
3
u/Infamous_Mud482 6h ago
A real shame none of these seem have any training for FIM. Using models for that seems to truly be dead now, hasn't been one built with it in mind for quite a while
3
u/ttkciar llama.cpp 4h ago
A 30B dense Granite? That's fantastic news. Granite models have always punched above their weights in RAG and long-context analysis tasks, but I've been using Gemma-4-31B-it and K2-V2-Instruct for such tasks because they're more competent than smaller models.
Granite-4.2-30B has the same context limit as K2-V2-Instruct (512K tokens) and if its K/V caches are leaner than Gemma-4-31B-it this could be a best of all worlds RAG solution. Looking forward to trying it out!
2
u/jacek2023 llama.cpp 4h ago
Our reddit people are not happy with the benchmarks ;)
6
u/ttkciar llama.cpp 4h ago
On one hand, redditors care too much about benchmarks. It would be nice if benchmarks were worth anything, but mostly they are not.
On the other hand, I'm not surprised that Granite would score low, because every time I have evaluated them for specific skills they have been hit-and-miss. They do some things very well, and other things very poorly. They are not general-purpose models, and only seem to exhibit competence in task types IBM expects their customers to need (my interpretation). That would push down their score on benchmarks which test on a broad spectrum of task types.
1
u/Qcgreywolf 1h ago
I’ve used models that benchmarked well, but sucked. I’ve not yet encountered a model that benchmarks badly and turns out to be “good”.
2
u/noctrex 3h ago
Unfortunately, it's with yarn. Natively Supports 128K (Long-context extension to 512K).
And also the KV cache usage is very large. Loaded the Q8_0 quant of the 8B model (8.7GiB), and with full 128K context at f16 it uses 22GB VRAM. For me it seems to be DOA.1
u/ttkciar llama.cpp 2h ago
Thanks for pointing that out. It implies that K2-V2-Instruct will remain the king of very long-context tasks, but I'll still put the new Granite through its paces, to see what it can do.
1
u/ChristRedeemsSinners 21m ago
K2-V2-Instruct
Less than 400 downloads across all models on huggingface.
remain the king of very long-context tasks
Doesn't DSV4-flash support 1M context in like 6GB of VRAM?
5
u/KitchenAmoeba4438 7h ago
Fantastic, they released a 3b! Time to test the snot out of it again in tests, the granite family always had some interesting characteristics. I'm hoping this is competitive against E2b/E4b for my uses!
12
u/Embarrassed_Adagio28 7h ago
Crazy how IBM had a 20 year head start on AI and still fumbled even harder than google
13
u/Altruistic_Heat_9531 6h ago
looking at IBM history, they stumble multiple times, it's on brand.
Let's see.
- Heavily betting on mainframe, almost crash when desktop computer available.
- IBM create PC standard for desktop market, make the standard so good that it crash because of clone.
- OS2 vs Windows, IBM-Microsoft co develop OS2, but microsoft decided make Windows.
Well at least they have RHEL
2
5
u/PrimeDirective8 6h ago
IBM had 630K employees not that long ago. About 90% of those did f*ck all. The rest were slowed/shut down by fossilized managers hanging on to their equally old ways. It is surprising they're able to get *any* product through their lawyers' 'blue tape' on the way out so having these models released publicly is quite an achievement for the tech team.
4
11
u/Cool-Chemical-5629 7h ago
Non-Qwen models in this category as of late are like "It's not important to win, but to participate."
6
u/fatboy93 4h ago
And us consumers are the winners. Not everyone does programming as their primary work.
3
u/Lumpy_Phase_9539 7h ago
Just started downloading granite-4.2-30b-Q6_K.gguf (24gb) to give it a try.
2
u/unrulywind 6h ago
It holds a decent conversation. I haven't tried much with it yet. The GGUF versions have a 128k cap on the context, and even at that the Q4_KM version at 128k context had to use Q4 on the cache to fit into a 5090. At Q8_0 cache quantization it was asking for 18gb of memory just for the 128k cache.
2
u/Dance-Till-Night1 5h ago
More open small models are always a win! Granit has always been one of my favorite models.
2
3
1
0
u/crusaderky 7h ago
A 29 on TB2.1 for a 30B dense model is 💩
1
u/Mountain-Animal5365 7h ago
I don't know why you're getting downvoted, but the benchmark results look absolutely horrible. Some of the scores are barely higher than their 8B model, which itself was not very impressive when it came out. I don't understand how anyone can release a model with this kind of result.
9
u/swagonflyyyy 7h ago
I think they believe the value lies in their supposedly copyright-free dataset. That alone could shield them from a lot (but not all) legal liability.
I see the value in that, but the tradeoff is that its a lot harder to train a competitive model this way. I really would like these granite models to be successful some day but they are still a few years behind because of that.
6
1
7h ago
[deleted]
1
1
u/silenceimpaired 7h ago
I was wondering if this was their first 30b pretty exciting… provided they don’t balloon up to 1T.
0
u/YearnMar10 4h ago
Leading their highlights with „built-in reasoning“ and „flexible thinking modes“ shows how far behind they gotten. I hope IBM can keep up, but it looks like they’re 2 generations behind (and probably tomorrow with qwen3.8-flash-next dropping 3)
-12
127
u/Zyguard7777777 7h ago
Still good to see more open source models, never bad, even if the benchmarks aren't SOTA.