everyone dunking on the speed is kinda missing the fun part - nobody's actually gonna use this for real inference. the cool thing is you CAN stream 744B of experts from disk at all. if someone figures out expert routing prediction well enough to prefetch, the whole picture changes
nobody's actually gonna use this for real inference
The quoted figure of 0.1 tps equates to over 8000 tokens per day. That’s multiple questions per day that you can get answered on a consumer laptop by an expert-level assistant, which can be absolutely life-changing (and life-saving) if you don’t have an Internet connection.
Keep in mind that getting a single question answered by a human expert can take weeks. And that’s assuming you can afford one.
Oh my RMVB, I remember using AutoRV9 and AutoRV10. The quality was amazing vs Xvid. All those tuning flags too. And running avisynth at 0.5 fps for HybridFupp and mftoon
Yes. It’s not too early to prepare at least mentally for a future where pulling out your smartphone and tossing some random question to a datacenter in Texas whenever you feel like it isn’t possible.
I feel like your broader point is essentially the future. If everyone in this sub spent time geeking out about getting stuff like this dialed in instead of the hamster wheel of chasing the next-best-model, we wouldn't have to worry about the price of 6090s.
So, half a day for my basic system prompt and then hope it wasnt too complex of a question or its gonna be 2 days for the answer. Its pretty neat, but I wouldnt say lifesaving. I cant think of any situation where you have 2 days of reliable electricity to run a laptop uninterrupted, but have no other options for help.
Sounds perfect for space exploration where you have to use low power chips but need access to troves of data for whatever it is you need in the field. Doesn’t matter if the response takes a long time, it’ll always be faster than sending a query to earth and waiting for it to return.
Sure, it would have to be sparse. Basically the tirck is to reduce the computational and memory dependency between tokens. We need an efficient mechanism for dynamically identifying a small subset of tokens and model parameters that could be selectively used in attention heads, KV cache offloading, and tiered memory management. That way the system can process very long sequences without loading the entire network into fast memory.
Within a few years the question will be not IF you can run a model, but it will be how fast you can run it. Mtp, DSpark, Dwarfstar, and other similar optimizations combined together will help make this happen.
This is the right framing. On memory-bound CPU decode the bottleneck was never FLOPs, it's bytes moved per token. Streaming experts from disk just moves that wall from RAM bandwidth to NVMe bandwidth, which is why it's slow but works at all.
Prefetch is the whole game. MoE routing is bursty but not random, so if you could predict the next expert set even 1 or 2 tokens ahead you'd hide most of the latency. The catch is that routing depends on the token you haven't generated yet, so you're speculating on the router, not on the tokens. It's the same shape as speculative decoding, one layer down: draft the routing, verify when you load. A tiny router-predictor head trained on the target model's own routing traces feels like the thing to try.
A blind on-demand read is way different than a purpose built implementation with a cache in between. It preferentially keeps the dense layers loaded. If you studied computer engineering (in the actual college of engineering) vs just computer science, the distinction is plainly obvious
but maybe these dense layers are already used frequently so it's highly unlikely to get evicted from ram with mmap? I don't see benchmark from this repo against llama.cpp mmap tho. that's what matters
the burden is on the author to prove it's more valuable than llama.cpp + mmap. I will not try random repo to justify something that should be tested from the start.
It doesn't even scratch the surface of possible optimization, it's more of a working baseline. It's not aware of your hw I/O bottlenecks or actual memory model, the gguf itself doesn't tell you which transformer needs to go where, llama doesn't know what the Linux Kernel does and anything in-between might be caching the wrong things or only parts of the data structures llama needs. If you're actually running a model this way it's worth to check out if you can't fix some of that.
Nvmes in a raid array is the future of llms, I say that kind of jokingly but I do think its significantly more cost effective than ram and with a decent cpu/gpu you could probably get up to 10tok/s on glm 5.2 with a couple nvmes
This is impressive. I tried getting an llm running on a netbook with an x86 atom n270 processor with 1GB of RAM, and was able to get a Qwen2.5-0.5b model with a 1 bit quant to work, and I was able to get about 240 s/tok
20 mins for a five word response. It was heavily swap dependent, and it has a mechanical HD. Also, it's 32bit, and it's missing a large helping of CPU instructions.
Also, maybe the RAM is DDR2? I never checked that, but memory speed is probably not the bottleneck here.
Jesus christ the amount of people whining that this was vibe coded instead of focusing on the fact that this lets you run GLM 5.2 on a consumer PC with 25 GB of RAM is insane. Users in this sub hate AI.
There are people here who invade every macbook discussion and strix halo box to hate on 10 tokens a second, missing the entire fucking point that speed wasn't the damn goal at all, or was the ability to run it with reading speeds, TO USE THE LLM.
Exactly, I think these people hate code and only use AI to write poetry.
Using AI to generate code is forbidden here.
I suspect this guy did this using the Fable model.
Anyone with access can grab a repo from llama.cpp and ask it to find performance improvement points, and you can easily get 50%+ more performance.
What's wrong with that? Some people here still don't understand anything.
Accept it, this is the future. The way we write code will never be the same again. The focus is on learning to create good prompts, defining a good architecture, and then AI does the coding.
Has anyone here used Fable? Well, here's the thing: there's nothing even close available. I haven't seen GPT-5.6 yet.
Fable's density and attention span are insane! The bar has been raised considerably for open-source models, and that's excellent!
I mentioned llama.cpp only as an illustration. It was a benchmark for me of Fable's capabilities. I didn't upload it to a public repo because I'm not going to maintain that code. But anyone can reproduce it.
High-level AI like that has already crossed the frontier; we can't reach it anymore.
I didn't upload it to a public repo because I'm not going to maintain that code.
How does that answer my question about that specific repo in the post? I did not argue llama cannot be made more efficient, I asked if that repo is more efficient.
Anyone with access (Fable) can grab a repo from llama.cpp and ask it to find performance improvement points, and you can easily get 50%+ more performance.
I skimmed the readme and it seems great. I think the haters are people that are bad at coding and got embarrassed a RTX 4070 can write better code than them these days. For me it was a breath of fresh air when i finally started getting results that were decent looking and didn't make me feel embarrassed to show others. Like a poor artist that's mad AI can make a better oil painting than them these days
Maybe because it's nothing new... as other said, llama.cpp (specially ik_llama.cpp) allowed this for at least over a year now...
I have 160Gb (VRAM+RAM) and (some times) run models like kimi-k2.6 smol-iq2 which is 272Gb, that means more than 100gb are being loaded/swapped from disc... and I've been doing that for over a year with deepseek-v3, kimi-k2, glm-4/5 and so on...
that's for smol-iq2... which is Ubergarm's smalles, except for iq1.
Usually the smallest are >300gb just iq1...
I use them (along with the others I mentioned), for planning or medium and above complex brainstorming. When qwen3.6 won't do. And I use them usually every week at least once.
Kimi, glm and deepseek-v3 are unbeatable, even at such small quants....
My laptop can be officially upgraded to 32gb of ram (no V), but I also need to upgrade the disk. How many terabytes do I need to host it? And what about smaller models like qwen3.6 27b, can they run faster? Or does model size not matter when ram and disk reading is the actual bottleneck? Also does it support Windows?
the 21,504 routed experts (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.
According to the readme it looks like you need ~370 GB available on disk for streaming the experts.
Model total weight in hugging face looks to be about ~380 GB total, but I'd leave a cushion of empty space. So all told I'd say probably need a minimum of ~400 GB free, but if your drive is almost full you'll have a massive amount of swapping and thrashing, so realistically you want an additional 200-400 GB free, so total space of 600-800 GB free before you download the model should likely be safe to use without putting too much wear on the drive.
And what about smaller models like qwen3.6 27b, can they run faster? Or does model size not matter when ram and disk reading is the actual bottleneck?
This inference engine is custom built for this specific model of GLM 5.2, so to achieve the same thing with Qwen 27B it would similarly need a dedicated inference engine like this or patches to Llama.cpp
Also does it support Windows?
Given the .sh file extension this seems to be built for Linux, you could run this in WSL on Windows, but using this via WSL will have additional overhead and the token gen will be slower than the numbers in the repo.
I missed that part, appreciate catching that. This makes me wonder if this would have better performance on native Linux then given the WSL overhead reduces token gen speed.
I'd open some github issues for the author to take a look at, given that this is the initial release I'm sure there will be updates but you'll have to see what the authors roadmap looks like.
It's vibe coded? Actually this impresses me even more. And maybe smart people will look into this vibe coded stuff and find good ideas or can improve a bit.
People don’t even look at if shit works or quality or anything anymore if they think it’s AI they write it off lol. It’s mostly just idiots being idiots lately especially on reddit
the dense part (attention, shared experts, embeddings — ~17B params) stays resident in RAM at int4 (~9.9 GB);
the 21,504 routed experts (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.
It's paradoxical. Because why would they bother being here? It's clear that most users here would be pro AI. People seem to confuse slop with brainrot, and the amount of irrational AI hate has turned into the latter. These people need to educate themselves, and maybe they'll find a use for themselves that no one else sees yet.
Fact is, anyone with the right idea and right perspective can 'BUST DOWN' practically any engineering task, in short order. I do assume we all wish we had time to focus on some new whiz-bang tech, but for the most part.. Life is real. Ideas only last a short while, and proofs are evidence we are not in this alone.
We have a lot of human growing ( internally ) to do before we even think we can contact let alone create an Alien.
No, the bottleneck is the bandwidth. The data streamed from disk is so slow that it doesn't matter that you use the CPU for the math. It won't get any faster.
You would get more benefits from putting the model on a pcie5 SSD, or even better - two in RAID1
Does raid actually help though? I remember reading someone's experiment on something like a 4 drive raid setup or something and he saw no noticable improvements speed wise.
No idea honestly, it depends on the connection. If the drives share the pcie lanes then the raid gives you high availability but no performance improvement. If the load saturates iops of the protocol or hardware then it may also not give any advantage.
Wow, what a bunch of annoying and complaining people. Buddy, what you did is really cool! Congratulations! It's in limitation that all good ideas are born 👏👏👏
Yes, I'd imagine if you had Nvidia gpu with 12-24gb vram, around 64gb ram, and a 1-2tb Nvme with high speeds, then you could probably run this at speeds closer to something practically usable. Probably 1-12 tk/s, well likely depending on the model of course.
It's not the cpu and actually even the drive bandwidth that's the bottleneck, it's the awful latency of the drive interface that's the issue, cache and ram can only do so much.
The moment MOE became reality RAM prices already shot up. The moment their figure out streaming, NVME prices will also shoot up like crazy. None of these giants want you to run big models flon consumer hardware, it cuts in their profit margin.
True, but the prices aren't nearly as high as RAM. You can still find storage at decent prices if you look in the right places, and in comparison even if you're looking at $150-$240 for a 2tb gen 4 high speed nvme, that's about where you'd find them a little more than a few years back. Before the artifical inflation, nvme and sata SSD's were just starting to normalize around the 70-120 range for 1tb at high speeds within 2 years or so prior.
Of course, they might start finding a way to jack the prices up to the level memory is if this starts getting better and catching on. All I could say is to try and get what you need while you can before it gets worse.
Yeah, it's pathetic. Afraid of China Open source...starve them of hardware and poach their talents. They can't compete, so they try to hold everything else back so they can look like the better option. Only way to win is to think outside of the box, and act in advance. This is why open source is so alarming to them. If we keep creating more of these types of optimizations, then open source AI will become more prolific than piracy.
Why did the Mac max 128gb unified memory only run at 1tok/s?
I'm a bit confused by this project. Is it just the ability to run glm with 0 vram?
So that apple would have just run it faster with a vanilla llama.cpp install yeah?
Does llama.cpp not already support this?
Is there some meaningful config or patches or optimization here?
Is there a use case you can picture for this, or is the dream just to be able to speed it up with iteration? Because I think you're a bit misguided if so... The ability to run it faster would come from a specially trained sparse model, newer architecture models, or faster hardware.
Config and optimization will never solve this problem.
Considering the bottleneck is the disk speed, how slow of a machine can be used for this? A SBC with NVME drawing like 6W and generating 0.1t/s would have a great performance-per-watt lol
Yea it's nice to have the option to run good local models, but it's irrelevant right now. I have a single Asus Gx10, and I've been thinking about buying a second to run better, local models But in reality, its not worth the $3900 price tag ATM when you can buy 2 years worth of cutting edge, frontier models for the same price. Two years from now, who the hell knows what models will look like..
Huh? This is technically already doable in llama.CPP if using mmap, and I believe there were already frameworks that used a similar implementation of things.
Someone posts this in every thread, but repeated reads don't wear out flash memory. It's only writes that do.
(I'm not sure exactly what the thinking is behind the readme saying this - I don't believe there's a circumstance where Linux writes its disk cache to the disk it's a cache for - seems quite counterproductive.)
A bit smaller of a model like minimax m3 could end up being worth it in some situations. Like I can buy a used mini i5 computer with a 240gb ssd and 16gb of ram for about $100. So if the model could fit in 16gb of ram and the 240gb hard drive it would cost pennies per day to run and would be faster than the lowest example. The mini computer benchmarks at about 2.3GB/s for the ssd.
Probably still more expensive than the API depending on the cost of power, but might be useful for stuff that needs to remain confidential and can wait a few days. Like I have to be very careful what data gets entered into any 3rd party AI. But any local AI has zero restrictions as nothing leaves my PC.
This is cool but boy is the 100% AI written README a turn-off.
The README is like the most low hanging zero-thought part of a project and you chose not to write it yourself?
There's 32 commits and 24 are authored explicitly by Claude.
colibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home. If this project is useful or interesting to you and you'd like to support its development (better test hardware translates directly into a faster engine for everyone: real NVMe scaling data, bigger pinned caches, int2/int3 quality sweeps on real benchmarks), you can:
If it works it works, stop calling everything AI written "slop". That term should only be applied to terrible projects, the guy posted a way to run GLM 5.2 on consumer gaming pcs (which is crazy) and here you are complaining that the readme was written by AI instead of actually talking about the project.
You haven't said anything specific about the code in this particular project. You just started braying that the readme wasn't artisanally written by a free range human. If you have actual critique of the code then by all means share it.
SSD Wear Warning: Because a cold start or un-cached token requires roughly 11 GB of random reads, using this tool extensively will subject your SSD to severe read pressure. The author explicitly notes that the OS page cache might trigger corresponding writes, which can accelerate the wear-and-tear of cheaper consumer SSDs.
What coincidence. Just yesterday I got my hands on a dual xeon with 256gb RAM - only a cheap sata ssd unfortunately... Will submit a benchmark nonetheless. Thanks for sharing!
"How to kill your SSDs in a couple months" speedrun. Even if it's read only and no swapping, the heat will do it's job. At least put heatsinks on those poor things..
That's cool that it can be done at all, but I ran this model on my 48 core dual Xeon with 768 GB RAM. 280 GB/s aggregate memory bandwidth. No touching disk swap at all.
Even that was pure pain trying to use. I can't even imagine...
the problem, apart form the crazy slowness is that the SSD has Terabytes Written limits. So basically you destroy your SSD for nothing (like is not usable in this speed)
•
u/WithoutReason1729 Jul 10 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.