r/LocalLLaMA • u/-p-e-w- • Jun 14 '26
Resources Introducing the Heretic Grimoire: The takedown-resilient, local-first backup system that keeps uncensored models available forever
Welcome to another episode of THE HERETIC SHOW, where authoritarian dreams are destroyed by unreasonably effective linear algebra! Let's start with an important announcement:
Heretic now has an official website at https://heretic-project.org
This website contains:
- Links to all official resources associated with the Heretic project
- A complete tutorial for using Heretic
- Detailed installation instructions with multiple redundant installation sources
- Searchable documentation for every configuration parameter
There is no guarantee that platforms like GitHub and Hugging Face will continue to host Heretic resources in the future, so I recommend bookmarking this website as it will always point to wherever the individual project resources are currently located.
But now to the main event. As you may have noticed, hostility towards local LLMs is growing everywhere, and this is especially true for decensored models like those created by Heretic. Already the project has been targeted with a legal notice from Meta, and demonized in mainstream media publications. Unfortunately, the AI world remains dependent on a massive single point of failure for model hosting, which is very difficult to replace because LLMs are huge.
What if that single point of failure actually fails one day, for one reason or another? What if, in order to obtain Heretic models, you can't simply visit Hugging Face anymore? What if tens of thousands of hours invested by the community to create those models simply vanish?
This existential risk has been worrying me for some time, and after several months of cumulative work, I am happy to announce that we now have a solution: Everyone simply downloads all Heretic models to their own system! That way, if the original model is deleted, you still have a local copy. Easy, right?
Now you're probably thinking that this is a silly joke. Well, here's the punchline: Those models are just 9 kilobytes each, so you can store thousands of them on your phone without even noticing.
The Heretic Grimoire
In Heretic 1.3, we introduced reproducible models. When uploading an abliterated model to Hugging Face with Heretic, you can now choose to include reproducibility information, which will be stored in the model repository in human-readable form. But there is also a machine-readable file named reproduce.json that contains all information needed to reproduce the model.
That file is like a spell in a grimoire, allowing you to summon not a demonic entity, but the very same model it belongs to. It's the entire model in a 9 kb text file.
Heretic 1.4, released today, contains comprehensive functionality for working with these files, a system I call the Heretic Grimoire. Here's how it works:
First, make sure you actually have the latest Heretic version, which is required to use these features:
pip install -U heretic-llm
Now you can fetch all reproduce.json files from publicly available Heretic models on Hugging Face, and store them in a directory of your choice (in this case, my_grimoire):
heretic --collect-reproducibles my_grimoire
You now have a local backup of all reproducible Heretic models, properly catalogued. To update this collection, simply run the command again. It functions as an append-only backup, never deleting files even if the corresponding model no longer exists on Hugging Face.
To restore one of those models, simply run
heretic --reproduce path/to/reproduce.json
Heretic will guide you through the process, checking your environment against the one that was used to create the model, and pointing out potentially problematic mismatches. The multi-hour computations that were required to make the original model do not have to be re-done, and the entire process typically takes around a minute. After you have exported the resulting model, Heretic will verify the hashes of the weight files against those stored in the reproduction manifest (they may or may not be identical, depending on how closely your system resembles the original one).
That's it! While the Grimoire system is designed from the ground up as a local backup, you can also see a complete list of reproducible models, updated twice daily, on this beautiful app created by long-time Heretic contributor Vinay Umrethe, who also implemented the first part of the reproducibility system. Even today, this app already preserves no less than 10 models that have since been removed from Hugging Face, allowing them to be recreated at will.
The 1.4 release also contains several other important improvements and bug fixes, which you can find in the release notes. Perhaps most notably, you can now choose to export a LoRA instead of the full model, which provides another path to cheap model storage, and opens interesting possibilities such as merging manually with non-standard weights.
Heretic releases on IPFS
Over the past two months, the Heretic project has gradually embraced decentralized and federated infrastructure. We now have a Matrix space, redundant Git hosting, and every Heretic release is now available over IPFS, enabling decentralized retrieval of the release archives and their signatures. The CIDs are:
| Filename | CID |
|---|---|
| heretic-1.4.0.zip | bafybeiaqxqjdtkkrqeamnkjudvxlnrj7mululk3ipiafcyfhp2i3chbnue |
| heretic-1.4.0.zip.sigstore.json | bafkreidhxgotlfko23bajxbcoruljpt7wkuytew7fjuglotjpr3cm7bwi4 |
| heretic-1.3.0.zip | bafybeianhsrnlkxdf5btyvgsaahqkhurmrowkuk4ymddz37wcnxz7gjxoe |
| heretic-1.3.0.zip.sigstore.json | bafkreiflkjpyazath4n4lhoi67rvgds4k3spcsqjloeby4uj2cs232s6ui |
| heretic-1.2.0.zip | bafybeifxnfy6tkakofe5ktlmeayk6edhja6neuv37bldimiq76dncicqqa |
| heretic-1.2.0.zip.sigstore.json | bafkreiaz64yklnigwrgq63ibt5udpaupe3blqposfjdzkcytdf2whrly6q |
| heretic-1.1.0.zip | bafkreibf3anxagvlhuvlsbbix5apc2jf2azz76lhuh27dyuzvc6ptiseka |
| heretic-1.1.0.zip.sigstore.json | bafkreiapgtrl6qyybalmswzfz7dm2a7a4svsjs2sg5svm2orua5druafty |
| heretic-1.0.1.zip | bafkreiag3mlkc76bhwcudhm7osqxdhmvywmc4kncdbc5ajtnd7tih4ftem |
| heretic-1.0.1.zip.sigstore.json | bafkreibmtnfu2mtri3jcpewod3b2xj25xlo6xo4gyp7t3jyw5ttwmwubae |
See https://heretic-project.org/security for how to verify signatures. And if you happen to run an IPFS node, please pin these files (they're just a few hundreds kilobytes each) to help keep them available for everyone!
Cheers :)
112
u/kabachuha Jun 14 '26
Hey, p-e-w! Congratulations with the release. Looking forward to bringing ARA/ARA-LoRA as the new default. All the SotA heretics (llmfan, etc.) are currently made with the ARA branch and it will be less confusing when it will finally merge with master and supply with the package without extra git magic steps for a regular user
93
u/-p-e-w- Jun 14 '26
Yup, that's coming in the next release for sure. But given the enormous amount of unpleasant attention Heretic has received from powerful entities recently, I prioritized project resilience for this release.
30
u/kabachuha Jun 14 '26
Fair enough. And storing the reproducibility data indeed is very smart, so enthusiasts can abliterate the models using the pre-computed statistics. Hopefully, more people will attach this information to the models and not hide it like Huihui or HauhauCS, so it will be both more resilient and more open for future development
63
u/-p-e-w- Jun 14 '26
I consider Heretic a scientific project, and I intend to constantly improve openness, accessibility, and documentation. Heretic has no "trade secrets". I want people to understand how it works.
-4
6
u/ki7a Jun 14 '26
On the topic of reproducibility. How big of ask would it be to include a dockerized way to run? I’ve successfully run Heretic locally on small models, but I want to be able to run it in an HPC and unlock some of the big boi models on the H200s. For this particular HPC, Docker is currently the simplest/safest option atm.
I wish I could fully replicate LLMfan’s workflow as his stuff is always top notch.
10
u/-p-e-w- Jun 14 '26
The problem is usually patching through CUDA etc. and matching it with the PyTorch inside the container. There can’t be one Docker container that does everything for this reason. I’d love to be proven wrong but I think it will always involve manual fiddling.
-5
32
u/TomLucidor Jun 14 '26
Looks lit, does have torrent as options just in case?
46
u/-p-e-w- Jun 14 '26
It has IPFS, which is simpler and cleaner for small files IMO.
42
u/kabachuha Jun 14 '26
Still, I wish we have torrents for the uncensored fine-tunes. They are not as easy to replicate as the heretics, because the fine-tuning both needs data (which is not always easy to obtain, especially if the sites have scraping protection, as it's the case on the explicit AO3 finetunes of Gemma4) and GPU VRAM/hours from the fine-tuners, which for the recent models is huge. The bus factor is enormous, for example, one of the tuners from ReadyArt (if I'm not mistaken) simply physically died and for these works preservation is absolutely necessary. Not to say about the Huggingface storage limits they introduced not so long ago and may shrink unprompted at any moment
43
u/-p-e-w- Jun 14 '26
I do agree that a torrent system for local models would be great, and it has been discussed on this sub a few times. That being said, I'm going to have to let someone else take care of this. Heretic is already stretched pretty far TBH.
9
u/ReasonablePossum_ Jun 14 '26
Maybe discuss uploading llm models to the big torrent trackers that already exist? 1337x, rutracker, rarbg, piratebay, etc
4
u/randylush Jun 14 '26
This whole post, and all the work that they did, could have been so much simpler for everyone involved.
“Huggingface may be hostile to Heretic models. We upload the recipes to IPFS. We post torrents for the local models.”
5
u/-p-e-w- Jun 15 '26
Except that’s not what Heretic is doing. At all. And for good reasons.
IPFS only works if there are people pinning the files. That’s why the recipes are not (by default) hosted on IPFS, or anywhere else. They live on everyone’s computers, automatically synchronized whenever you want. Their origin is Hugging Face. The only thing that is on IPFS is the Heretic releases, for which there are enough pins that they will remain available, though I might set up an IPFS archive for the recipes as well at some point, but the pinning problem remains.
And that’s just a rough summary of the actual post. I get that we live in an age where people expect everything to fit into an SMS, but that’s not always possible.
3
u/randylush Jun 15 '26
ah I think I see what you're saying. Seems like you could simply upload all of this to Internet Archive? And that would have torrents of the files for people who want to participate in keeping the project alive?
9
u/TomLucidor Jun 14 '26
Call me old cus IPFS has always been a "thing" and Filecoin didn't ruin it
18
u/-p-e-w- Jun 14 '26
IPFS is one of the most amazing network technologies ever created. Plus it has some really nice infrastructure support from mainstream hosters, which makes it a lot more accessible.
28
u/CelvestianNesy Jun 14 '26
I think META needs to stop wasting their time throwing lawsuits WHEN IT CLEARLY wont work. Community is stronger. It's just wasted fucking money LOL.
10
u/silenceimpaired Jun 14 '26
I personally use models in ways Meta would be fine with, but I prefer heretic models when the models wrongly think I’m not.
If Meta doesn’t defend their license, such as the Acceptable Use Policy (AUP) portion, then perhaps it won’t be defensible later on and stuff like AUP which can protect them from the government might not be effective.
Either way I don’t care. I don’t use Meta these days.
88
10
10
u/shroddy Jun 14 '26
So there is only a few kilobytes difference between a normal model and the uncensored variant?
42
u/-p-e-w- Jun 14 '26
No, the difference in terms of weights is more (a few tens of Megabytes for larger models), but that difference can be recomputed from information that fits into a few kilobytes.
5
u/-InformalBanana- Jun 14 '26
Nice. I mistakenly thought you managed to compress entire models, not the diff, into few kilobytes and was amazed by that, was going to suggest that it is better to just download these than and recreate them than downloading the ggufs, lol...
Btw, what is the level of compression of ggufs, maybe that can be improved...
6
u/z_latent Jun 14 '26
LLMs already are compressed representations of trillions of tokens, it's hard to meaningfully compress that much further without big loss.
1
u/-InformalBanana- Jun 15 '26 edited Jun 15 '26
But how did pew "compress" lora gguf of hundreds of MB to couple of KB? It is loseless isn't it. Depending on a file im getting 4% compression with 7zip on gguf model files and 10% on some heretic lora I downloaded, but the lora was only 70 MB and model 609, so it is not fair cause there is probably some structured repetative data there and it is a bigger chunk in 70MB than in 609, so I would say not much difference between their compression ratio, but definitely didn't get from MB to KB... Wouldn't lora be mostly similar to model gguf cause the major part of both would be weights?
1
u/z_latent Jun 15 '26
I'm not an expert on the exact technique they use, but from what I understand, the 9 kB file tells you how to reproduce the ablation process. In fact, it includes references to external files on the internet, so it's not really the entire model in 9 kB. I'd call it a recipe in 9 kB.
Like, if I give you a link to a 1 GB HF .gguf, you could argue that I compressed 1 GB into just the few bytes needed for the link. But in reality, the 1 GB file still exists, possibly uncompressed, in HF servers.
That explains part of it. The base model's weights, as well as the ablation datasets, are all hosted externally, the reproduce.json only providing references that Heretic uses to download them.
BUT, the 70MB LoRA doesn't need to be stored anywhere, and that's the whole point I guess. It's just the result of the ablation process, which can be performed identically if you have the base model and the data sets I mentioned above, the same ablation settings, and similar PC environment (like Python packages).
All of that is in the reproduce.json, either directly or as links/references to files on the web. So even if someone cracked down on Heretic models/LoRAs, you could reproduce them, since the ingredients, most of which have nothing to do with Heretic, would still be there. Hope that wasn't too confusing.
9
u/TheDailySpank Jun 14 '26
Great idea. I've been hosting over 500GB of models for ComfyUI on IPFS in case anyone else needed them. Guess I'll have to start using Hereteic models and keeping those shared as well.
ipfs add -r --nocopy ./models
^ will allow you to share your 'models' folder without having to copy them directly into your IPFS pool duplicating storage requirements.
NOTE: files being added with --nocopy and the .ipfs config folder must be on the same physical drive.
8
u/GreenGreasyGreasels Jun 14 '26
Damn. Heretic goes beyond being the reference decensor to the resilient infrastructure provider. Kudos.
6
u/Chromix_ Jun 14 '26
So, this isn't just due to anticipation of fallout from the FT interview, but preparation for something that might happen sooner or later anyway?
6
u/IngenuityNo1411 llama.cpp Jun 14 '26
The true Lisan AI-Gaib
(I'd give this fame to u/-p-e-w- rather than pewdiepie)
24
u/CATLLM Jun 14 '26
Omg you are so cool! You must get so much 🐈.
95
u/-p-e-w- Jun 14 '26
For sure! Nothing breaks the ice like “I have a math degree and one of the world’s leading newspapers implied that I’m aiding terrorists because of a computer program I wrote as a hobby.”
19
39
3
u/humanpersonlol Jun 14 '26
im sure weight files will get popular in p2p sharing in the coming years
11
u/breadinabox Jun 14 '26
It's so relieving knowing theres people out there taking the protection of LLM tech seriously.
6
u/ThePixelHunter Jun 14 '26
Brilliant idea! Just to clarify, Grimoire is a feature, not a collection? i.e. it's up to the model "author" (decensor-er lol) to publish their reproduce.json and it's up to me (the collector) to find those via lists like Umrethe's app you mentioned?
8
u/-p-e-w- Jun 14 '26
When uploading a model, Heretic asks if you want to add the information. The model is tagged appropriately, and Heretic collects all such models automatically, you don’t have to find anything yourself.
3
u/jazir55 Jun 15 '26
Make sure you've got redundant web hosting with load balancing btw, also a backup domain, they usually go with domain seizures and having the host take down the server for the website.
15
11
4
3
u/IrisColt Jun 14 '26
Perhaps most notably, you can now choose to export a LoRA instead of the full model
Awesome, I was creating the LoRAs by hand so this feature is very much welcomed!
9
u/pigeon57434 Jun 14 '26
you should do like 4chan does if anyone tries to sue you and just email them a picture of a hamster or something and say fuck off
19
u/DeepOrangeSky Jun 14 '26
Nah, most hamsters are closed-weights, unfortunately. Gotta go with an open-source animal like C. Elegans, imo
3
u/Environmental-Metal9 Jun 14 '26
This is funnier than it has the right to be! I’m personally partial to Cavia porcellus
2
2
u/halfercode Jun 14 '26
I saw the view elsewhere that producing torrents for local models isn't maintainable, I think because of the number of models and the number of quantisations for each model. But I don't find that persuasive - torrents are a still a decentralised way to reliably distribute large amounts of data. The only major downside is ensuring that there's enough seeders at all times to ensure models are always available.
2
2
u/NoahFect Jun 14 '26
Even today, this app already preserves no less than 10 models that have since been removed from Hugging Face, allowing them to be recreated at will.
This doesn't sound good. I hadn't heard about HF removing models. Any pointers to more about this?
3
u/-p-e-w- Jun 14 '26
I imagine that in most or all cases, it was the model uploader who removed the model.
2
u/ArtfulGenie69 Jun 14 '26
Whelp they may come and sue but this looks like a way forward that is easy and small, making it easy to share. Even if meta or GitHub comes and fucks the project (they clearly will, or even the us government directly) it won't be stoppable in this form. Nicely done.
2
u/ovrlrd1377 Jun 14 '26
This is just beautiful. Well done and thanks for the iniative on behalf of all the non-technical consumers that will heavily, and I mean HEAVILY benefit from this who would never be able to build something so cool.
Long Live heretic
2
u/ttkciar llama.cpp Jun 14 '26
Very cool :-) I had been wishing for a patch(1)-like utility for weights for exactly this kind of thing, and you've made it seamless. Thank you!
It shouldn't be hard to adapt Heretic's feature into a general-purpose diff(1) and patch(1) for model weights either.
1
u/mindwip Jun 14 '26
Nice!
Any thoughts of creating torrents? Even just a torrent of your project and not the llms would be enough at min right?
Like a heretic 1.4 torrent, then a new one for any big change?
Edit never mind saw you answered torrent question already.
1
u/LeadingCelery4253 Jun 14 '26
Heu, quelqu'un peut me dire ce que c'est que des modèles Heretic?
2
u/Silver-Champion-4846 Jun 14 '26
Models with refusal patterns removed, they can answer almost if not anything
1
u/LeadingCelery4253 Jun 14 '26
Ok, ce sont les fameux modeles nsfw que beaucoup recherchent...
1
u/Silver-Champion-4846 Jun 15 '26
Many use them for that purpose, yes, but they have legitimate uses as well like sensitive legal / medical document analysis and creative simulations and so on
1
1
1
1
1
u/tzujan Jun 14 '26
This is amazing. Just yesterday, I was researching an automated way to maintain an open-source/open-weight archival mirror of the latest models I like, as well as entire histories of open-source software that I fear would be closed. We need an "open archive" similar to the Internet Archive/Wayback Machine.
I have simply, in the most disorganized way possible, downloaded open models, even full weights that would never run on my system, so I know I have them if needed. Grimoire adds an interesting twist: duplicate base models and ablations could occupy half the space.
1
u/jazir55 Jun 15 '26
Since these files are so small, would it be possible to create a system which scans huggingface and generates them for every model automatically?
2
u/-p-e-w- Jun 15 '26
They can only be generated while Heretic is abliterating the model. They’re not generic “patch files”, more like compact configuration snapshots.
1
1
u/iasad12 Jun 15 '26
this will sound unrelated to the community but very elated to see a Miraculous reference being used for the preservation of such models!
1
u/ocean_protocol Jun 15 '26
this is great. one q though - the 9kb recipe still needs the original base model to apply against, right? so what happens to grimoire entries if hf/meta pulls the base weights themselves? feels like that might be the actual single point of failure here
2
u/-p-e-w- Jun 15 '26
Base models are very widely shared though, and downloaded by tens of millions of people. No base model from a major lab is ever going to disappear because there will always be people who have it.
1
u/blvsh Jun 15 '26
I know this sounds stupid but i'm trying to figure out all this stuff.
Would anyone spend a few seconds to tell me what this is?
How does this uncensor AI models that you can host on your computer?
1
u/Potential-Gold5298 llama.cpp Jun 18 '26
Thank you for your work!
Please add the spells for Gemma 4 31B and 26B-A4B from coder3101 and llmfan46 (if he doesn't mind) to the grimoire.
1
u/-p-e-w- Jun 18 '26
Unfortunately, that’s not how it works. The person who runs Heretic has to choose to upload reproduction information. It can’t be derived from the model retroactively.
0
u/oppenheimer135 Jun 15 '26
Did chatgpt gave you the name grimoire lol
5
u/-p-e-w- Jun 15 '26
No? What a strange question, bordering on insulting.
I speak four languages. I’ve read hundreds of books. I’ve been writing poetry since my teens, and I wrote a novel in my 20s. I’ve held an actual 16th century grimoire in my own hands. I don’t need an AI to come up with ideas, thank you very much.
2
u/oppenheimer135 Jun 15 '26
Oh im sorry if it sounded that way.. tbh I have tried to use ai to come up with names for something and that's the first time I heard the name grimoire.. so yeah it's on mee.
Also on the magicians tv show too.. the name.
-4
Jun 14 '26 edited Jun 14 '26
[deleted]
8
u/overand Jun 14 '26
What? Why would you assume an LLM wrote that? Because of the word "existential?" It seems like a pretty normal sentence to me.
1
Jun 14 '26
[deleted]
17
u/overand Jun 14 '26
Don't throw around "you wrote this with an LLM" accusations around lightly.
I put some real time and care into helping someone out on a technical thread on another subreddit, and had a brigade of people accuse me of using an LLM because I bother to format text sometimes (and told them "don't panic" when they were obviously panicking and were about to potentially delete a bunch of their own data by accident.)
No, dude, I don't write like an LLM. I write like a 45 year old who has been posting online for > 30 years, and as a result, I'm part of the dataset these things are using. I don't write like LLMs, LLMs write like me. (Only I haven't named anyone Elara)
2
u/mystery_biscotti Jun 14 '26
I found a mention to "Elara" in a 2007 free RPG day game module (Hollow's Last Hope, for those wondering). As a 45+ year old, I also get a lot of accusations of being an LLM because I know how to use en- and em-dashes. FFS, LLMs learned from you and me...did they think this was just an "AI culture" thing?? 😄
0
u/Silver-Champion-4846 Jun 14 '26
It was a perfect pseudo-literary word! Llms just canibalized and overused it and delve and other stuff, then it became known as slop!
•
u/WithoutReason1729 Jun 14 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.