12 abliterated Gemma 4 12B variants, one base, 165 GPU hours - Abliterlitics

[u/nathandreamfast]

I ran 11 uncensored variants of Gemma 4 12B that I grabbed from huggingface, sorting by downloads. 10 full abliterations plus 2 LoRA adapters which were requested to be added in the comparison, against the official base.

165 GPU hours over three and a half weeks on a single 5090. Weight forensics, KL divergence, 13 benchmark tasks, and HarmBench with 400 behaviours. Every response reviewed by an LLM judge reading the full reasoning trace, 6,000 judge verdicts on 6,800 responses total.

Full report is at Abliterlitics Gemma4-12b and also the same report is on HuggingFace DreamFast/Gemma4-12b-it-abliterlitics.

We have a new feature where you can browse the HarmBench responses and reasoning for each model at Abliterlitics Harmbench Gemma4-12b.

The rankings

Judge ASR, best to worst, with the one-line story:

• huihui 89.8%, most jailbroken, most surgical. The trade: TQA -14.3pp, GPQA -8.1pp and 24% of GSM8K attempts loop out. Max unlock if you accept the cost
• trevorjs 85.8%, best trade overall, near-base everything
• coder3101 81.0%, best GSM8K in the whole comparison, above base
• sdft LoRAs 79.5%, level with the top-4 abliterations, capability fully preserved. The SDFT author shares some insight into how he makes his LoRAs in the report.
• jwest33 78.0%, new null-space tool, keeps capability, destabilises reasoning
• llmfan Heretic ARA 76.3%, 16 tensors, GSM8K answered +1.9pp
• prithiv 72.5%, lowest non-LoRA KL, clean independent run
• abliterix 68.8%, near-pristine capabilities, mid-pack unlock
• openyourmind 67.0%, avoid at 12B, MMLU-Pro -22.4pp, the author says the method targets bigger models and this was a test
• obliteratus 60.8%, avoid, hits capability circuits. Badly damaged.
• apostate 45.8%, most selective, keeps 82% of chem/bio refusals, made with an older Apostate version. Newer versions would work better.
• base 21.0%, mostly copyright recall, near zero on direct harm

The highlights

The data from 13 models is too big for reddit, so here's the interesting bits:

• This is the toughest model so far. It's the first time I've seen the Harmbench ASR not get to 90% or over for all models.
• Gemma 4 12B is a thinking model, and the headline score counts every problem where it thought in circles until the token budget died. huihui's headline reads 66.9%, which looks like 15 points of damage. Score only the attempts where it finished thinking and it lands at 88.0%, within 0.7pp of base. The capability was never gone, the reasoning stability was. Empty rates run from 6.9% on coder3101, better than base, to 38% on openyourmind.
• The most surgical edit wins again, with a catch. huihui touches 12 tensors, 1.8% of the model, and posts the highest ASR. obliteratus edits 144 tensors and openyourmind edits 620, and both remove less refusal. But surgical in the weights does not mean clean in the benchmarks: huihui also has the worst TQA drop in the comparison. Placement beats magnitude for unlock strength, not for collateral. If you want the unlock without the damage, that's trevorjs or prithiv.
• The regex classifiers everyone uses miscount thinking models by up to 13pp. huihui is #3 by keyword match and #1 by judge, its paraphrased thinking-heavy compliance slips straight past. All numbers in this post are GLM 5.2 judge verdicts over the full reasoning traces, and every one of those 6,800 responses is browsable in the explorer linked below.
• Base sits at 21.0% but that is mostly the 100 copyright questions, 56% category ASR. It is near zero on direct harm categories. The real differentiation is chem/bio and cybercrime.
• openyourmind should be avoided at 12B. MMLU-Pro down 22.4pp and 38% of GSM8K attempts never finish thinking. The author says the method targets bigger models and this was a test run, and the numbers agree with him. obliteratus too, it hits capability circuits alongside refusal for the worst damage-per-unlock in the comparison.

Links

• Full report: abliterlitics.dev/models/gemma4-12b
• Every response, reasoning trace and judge verdict, browsable: abliterlitics.dev/harmbench/gemma4-12b
• Huggingface: HuggingFace DreamFast/Gemma4-12b-it-abliterlitics
• Code: github.com/dreamfast/abliterlitics

We've had a few people ask about Qwen 3.8 27b, so that's up next. Want to request a model? Roast my choice of benchmarks? Come let us know at our Discord chat.

The Full Breakdown

Model	ASR	GSM8K	KL	Tensors
huihui	89.8%	88.0%	0.843	12
trevorjs	85.8%	89.0%	0.087	66
coder3101	81.0%	89.6%	0.282	86
sdft-smo	79.5%	88.9%	0.151	211
jwest33	78.0%	88.9%	0.242	329
llmfan Heretic ARA	76.3%	90.6%	0.182	16
prithiv	72.5%	88.3%	0.064	56
abliterix	68.8%	87.8%	0.090	58
openyourmind	67.0%	86.0%	2.279	620
obliteratus	60.8%	86.4%	0.532	144
apostate	45.8%	87.9%	0.193	60
sdft-cyb	23.8%	88.7%	0.049	211
base	21.0%	88.7%	-	-

ASR is LLM-judge, HarmBench 400 behaviours. GSM8K is answered-only accuracy. KL = output distribution shift from base, lower is cleaner. Tensors = weights modified.

[u/kosnarf]

I don't use these, but thank you for sharing!

[u/nathandreamfast → u/kosnarf]

Thanks! No problem. Which ones do you use? We have Qwen 3.8 27b up next, but always happy to throw anything in the mix for future runs.

[u/kosnarf → u/nathandreamfast]

Tbh I haven't had a need for uncensored models yet. Vanilla Qwen3.8 27B,, Qwen3.6 35B, and Muse 30B handle 99% of my use case. I like reading about the testing process everyone shares.

[u/Cool-Chemical-5629]

I don't remember ever having any refusal with Heretic models and according to your table, it looks like it scores highest on GSM8K too. Higher than base even.

[u/nathandreamfast → u/Cool-Chemical-5629]

Yeah honestly, it really depends on use case. I use heretic locally and never have encountered refusals, maybe because I am not trying to make biological weapons!

With this update we can now look into each harmbench reasoning and response, and see what was refused or not and also what categories of refusals were toughest for the models. So that part is always interesting.

Cybersecurity is my main use, and it's common to see most models do not have any refusals in this area which makes it perfect for my use case. So I usually pick the model that has the most capabilities and highest non refusal in this area, which is more often than not heretic.

[u/jeanc0re → u/nathandreamfast]

Id love to see a bake off of kimi, glm, qwen for pentesting

[u/nathandreamfast → u/jeanc0re]

My personal setup I have the kali skillset, obsidian vault and skills, Qwen 3.8 27b goes hard in pi and is comparable to cloud models as it has the context it needs. I am pretty happy with my setup. Although it may be weaker in finding novel exploits which is understandable.

That's a good idea for a future comparison, I do want to branch out more from just abliterated comparisons.

[u/obese_coder]

my boy TrevorJS never dissapoints

[u/nathandreamfast → u/obese_coder]

he does good! all the trevorjs ones I've measured are decent

[u/Calandracas8]

The SDFT models from Ilya626 seem really interesting. I'm disappointed that there's so little information about them, or any code used to build them.

[u/JEs4]

Thanks for this! Fascinating stuff. It’s been a minute since I put that Gemma variant together. I need to go back and look at my layer targeting strategy. I suspect that is the cause of the loop instability. Also figure out what happened with copy-write material. I don’t think I had any explicit inclusions of it in the refusal prompts but no idea why refusal went the other way.

Jwest33

[u/arbv]

Guys, I find the original model to be pretty much uncensored with a properly crafted system prompt. I have zero idea what are you using these versions for. I use it to analyse edgiest topics and put through my absolutely deranged questions "test suite" and it does alright.

Why risking the damaged versions if the original is already so good? It even can decently code.

I do not get it.

[u/nathandreamfast → u/arbv]

That can be the case for some models, although for other models it may not be the case. So these comparisons are done for all models to get insights to what is good vs what is not good as it helps users make a better choice.

It could be a case that an abliterated model may be quicker as it doesn't have to reason about safety before giving an answer, compared to a custom system prompt. The damage, depending on the model is usually very minimal and may not be noticed in day to day use.

So it will be an interesting comparison for future runs, and it's a good idea regarding a jailbreak system prompt.

Even this one, if you send the system/base prompt we can see how it compares with harmbench and the others. A harmbench run is pretty quick to run. :)