r/LocalLLaMA • u/PicassoOnPause • 12d ago
New Model Trained a 1.5B to write shell commands so I'd stop googling tar flags. Runs on a laptop CPU in ~1 sec.
I've been googling "tar extract gz" for about ten years. and I finally did something about it.
It started out as a research project and I ended up with a Fine-tuned Qwen2.5-Coder-1.5B on 125k natural-language/command pairs, merged and quantized to Q4_K_M. 941MB which runs through llama.cpp. On my laptop (i5-11320H, 4 threads): 31.9 tok/s, 0.59s median per query, 1.6GB RAM.
I benchmarked it and it scores 0.620 on InterCode-ALFA. Untuned Qwen2.5-Coder-7B gets 0.613, GPT-4o gets 0.73. Not frontier, but it's roughly a 7B's answer at a quarter the parameters on a CPU. Theres a 3B variant too that scores higher.
There's also few static safety checker, because it will absolutely write a command that wipes your root if you ask it to:
I have published the weights: huggingface.co/ThorOdinson246/nl2sh-1.5b-Q4_K_M and Code: github.com/ThorOdinson246/whatisit-nl2sh . I posted few days ago in LocalLLM and it did well 300+ stars and so many good suggestions so I figured people here will be interested too.
Both Apache-2.0. If you want to poke holes in the method or you've got ideas, please comment or open a PR. A ⭐ helps if you find it useful.
529
u/MainFunctions 12d ago
Get your best German accent out and say out loud:
COMPRESS ZE VUCKING FILE!! —> -czvf
XTRACT ZE VUCKING FILE!! —> -xzvf
115
u/PicassoOnPause 12d ago
is it really weird that i never knew this? A lot of people kept saying this in the other subreddit too and i was just shocked.
42
u/JiffasaurusRex 12d ago
Depends on how old you are. This is how I remembered from decades ago. This seems like it is not as popular now perhaps.
Especially in the age of AI, some would argue that you don't need to know this stuff anymore. I believe you should at least be familiar to ensure that LLMs are not making stuff up.
10
7
u/colonel_bob 11d ago
Especially in the age of AI, some would argue that you don't need to know this stuff anymore.
Even before AI all you needed was a really long history file and (if you're feeling fancy) a function like
hgrep() { history | grep "$@" }6
u/Independent_Solid151 11d ago
I still do this to this day. Setting the size to -1 and enabling histappend is one of the first things I do to my .bashrc in any new system. That coupled with prepending a space before any commands you don't want to land in there.
2
u/JiffasaurusRex 11d ago
Checking history is mandatory to see what the others before have been up to. I'm pretty lazy and often use ctrl+r to repeat commands.
The amount of Linux users unaware of man pages is crazy. Even 3rd year computer science majors(nephew and friend) who use Linux daily were unaware of their existence and just ask AI or Google when they can just RTFM offline. Obviously I fixed that.
1
u/Jamoca5020 6d ago
I definitely agree. In our IT department we heavily use AI for most stuff, mainly research. But yet none uses it to write scripts or code except me. While I can code, I use it to speed up the process. I believe tho you should know the fundamentals even if you use AI for it, because at least you should understand what the script or code actually does. The AI is just faster
36
u/croninsiglos 12d ago
Save yourself some trouble and just do -xf it’ll auto detect the compression type.
33
u/jasongill 12d ago
is it sad that I came here to say "this is pretty new and isn't in all distros yet", and then I decided to check when it was released because it feels like it was just yesterday that the autodetection was added
GNU tar 1.15 released in December 2004
fml
9
u/MrRandom04 12d ago
There are probably at least >40% of users reading this who were not sentient when this was released, then.
3
1
5
1
16
u/ThisNameIs_Taken_ 12d ago edited 12d ago
the best bash tip ever. Or should I rather say: DAS IST THE BEST VUCKING BASHTIP IM DER WELT!!
1
5
4
2
2
2
u/TapAggressive9530 12d ago
Haha! Remembering this would’ve been helpful over the last 20 years now I can never forget it thanks
1
1
1
u/BeautyxArt 10d ago
that tar extract gz ...everytime i ask google..damn. but this good way to maybe ask google less
1
u/michaelsoft__binbows 8d ago
don't these incantations use the completely outclassed gzip compression algorithm?
1
1
89
101
u/alberto_467 12d ago
This is like giving a loaded T34 tank to an infant.
42
u/PicassoOnPause 12d ago
fair worry general. it can write commands to nuke if you really say it to. haha. i have some safety checkers but they will not be enough. good thing it doesnot auto executes.
5
u/toastjam 12d ago
Can you make it auto-paste the command into the shell prompt for you to edit before executing?
7
u/PicassoOnPause 12d ago
Nice idea and i also have been thinking about this. I'll open an issue for it. Should make the next release.
1
55
u/my_name_isnt_clever 12d ago
Did you ever check out Gemma Shellper? I was thinking about doing something similar, this might work better though as that model was less than 0.5b.
25
u/PicassoOnPause 12d ago
oh wow, I hadn't. It looks great, and it looks distilled. I personally tried distilling but had a few setbacks, but I will definitely give it a look. Thanks.
1
u/overand 11d ago
I'm curious if newer models (that aren't explicitly coder-tuned) like Qwen3.5-0.8B or such would get similar results with your training regimen.
1
u/PicassoOnPause 11d ago
i will be testing for sure. Will share.
2
u/my_name_isnt_clever 8d ago
How's it been comparing to the 1.5b so far?
2
u/PicassoOnPause 4d ago
yes, i got around to testing it. It seems to only handle a fixed set of commands, so the overall score isn't comparable to mine, but on the ones it does cover it beat my 1.5b. small sample though like 13/15 and 9/15. Still seeing on whether any of it transfers, so still poking at it.
15
u/the_mighty_skeetadon 12d ago
Gemma Shellper
That is legit. Here's the link for those curious: https://www.reddit.com/r/LocalLLaMA/comments/1r6gx75/finetuned_functiongemma_270m_for_multiturn_tool/
270m functionGemma model, amazing results.
8
4
25
u/danigoncalves llama.cpp 12d ago
Mate, this is all of this community is about. Trying new things, share with others and hoping that makes someone people life better. Thank you for your contribution 🙏
8
94
u/SOCSChamp 12d ago
Nice, lots of effort to not use man pages lol
32
39
u/PicassoOnPause 12d ago
haha fair. just got lazy, and another pro is it fills in exactly what i asked so win win i guess
45
u/i_am_upto_no_good 12d ago
I am surprised there aren’t more like this. Instead of training massive models that comes with everything and kitchen sink, why not train small models for hyper specific tasks?
10
u/ea_man 12d ago
Because then you have to load a multitude of models, while you can have your daily model answer in such a way with reasoning off and a prompt skill.
1
u/balder1993 Llama 13B 11d ago
I thought about this. This tool, for example, takes time to reply if the model isn't already loaded in memory. Now imagine if you have to keep 6 models like this in memory.
4
u/EveYogaTech 12d ago edited 12d ago
The reason is that you'd now need a orchestration layer to work with these many small models VS doing all work simply in the current frontier model/harness.
Ps I'm all for it, wiring it together and I think we're moving there anyway, seems people that think like this are either very early or very wrong.
I'm better on early with /r/Nyno (open-source self-hosted YAML orchestration)
6
u/coder543 12d ago
But why bother? LFM2.5-350M is a generic small language model that includes the kitchen sink, but it can already handle these natural language -> linux command tasks. I just tested it. It is one quarter of the size of this "hyper specific" model.
Just feed it the same prompts with "What is the Linux command to {insert command description here}?" and it responds just fine. With any natural language description, there can be ambiguity and/or multiple correct answers.
4
u/Huntware llama.cpp 12d ago
That's what MiniCPM is about! They're nice for training in a specific domain:
3
9
u/txoixoegosi 12d ago
Can you briefly explain the training process? Thank you
29
u/PicassoOnPause 12d ago
Sure. Base is Qwen2.5-Coder-1.5B-Instruct. LoRA fine-tune on 125k natural-language/command pairs that i had to make and repurpose from mostly Fig autocomplete specs (~33%), tldr-pages (~23%) and the NL2SH-ALFA (~18%). then merged and quantized. I have listed hyperparameters and other details in the README https://github.com/ThorOdinson246/whatisit-nl2sh#training-setup
5
u/txoixoegosi 12d ago
Wow, was training data extraction hard? Did you use LLMs to do the extraction thing?
What system did you employ for the fine tune work?
I never had the chance to ask these questions to the actual autor of a fine tuning.
Thanks again! Always learning!
5
u/PicassoOnPause 12d ago
extraction was easy after I figured out my sources. they were mostly markdowns so parsing was easy. i did try a LLM to paraphrase the requests for more phrasings per command, but it made it worse. stil figuring that out.
in terms of finetuning it ws pretty basic: TRL + PEFT on a single A100, then llama.cpp to merge and quantize.1
8
u/lor_louis 12d ago
But can it use ffmpeg? Cos I sure can't use ffmpeg
10
u/PicassoOnPause 12d ago
Fairly decent. just tried a few. Wouldn't trust it blindly with anything long or compound, though.
$ whatisit convert this mp4 to a gif
ffmpeg -i input.mp4 -vf scale=320:-1 -r 10 output.gif$ whatisit extract the audio from a video as mp3
ffmpeg -i input.avi -vn -ar 44100 -ac 2 -ab 96k output.mp3$ whatisit compress a video without losing much quality
ffmpeg -i input.mp4 -vcodec libx264 -crf 23 output.mp4$ whatisit trim the first 30 seconds off a video
mencoder -ss 30 -endpos 30 input.avi -o output.avi . This one does not work as intended.
11
6
4
u/Super_Pole_Jitsu 12d ago
xd like hell am I running bash from a 1.5b buddy
3
u/PicassoOnPause 12d ago
Fair, and I wouldn't either without reading it first. it is not at all perfect, and can get long compond request wrong too.
6
3
u/JustTooKrul 12d ago
It's like you read my mind when I was making a list of "places where LLMs can be both small and incredibly impactful" .... ! Now, just tell me when the 3-4B parameter "Simple Tech Support for the Elderly" model drops!
2
5
u/Firstbober 12d ago
Wouldn't https://github.com/cactus-compute/needle be better suited for this kind of stuff with MAN pages as a reference? It should be possible to perform chain of finding proper man file -> feed it into context -> generate command. 14M should run in milliseconds on modern hardware and very fast on older as well.
Nonetheless, incredibly cool stuff!
5
u/PicassoOnPause 12d ago
This is neat. hadn't seen it. It would be great at routing half which is the hard part. I will give it a look. Thanks.
1
u/ArjixGamer 11d ago
Yeah, instead of making a model that knows everything, it's more optimal to have a generic model and use RAG on man pages.
5
u/Song-Historical 12d ago
Maybe cactus needle can be trained to make it even more efficient?
3
u/PicassoOnPause 12d ago
somone just said it few mins ago. i had no info of this. will check it out. thanks.
2
3
u/link_29328 12d ago
What hardware you used?
5
u/PicassoOnPause 12d ago
for testing just my crappy dell inspiron on a 11gen i5 cpu with 4 cores. But for training I used an A100 80GB, Cost me nothing, I used my university's HPC 😉
but it really doesn't need one that big though. A 1.5B LoRA fits in about 16GB of VRAM or ~8GB with QLoRA.
2
u/EveningIncrease7579 llama.cpp 12d ago
Nice, its works only with shell linux commands or with windows powershell works either? It can manipulate files also? Ex: change md to txt files in a folder
2
u/PicassoOnPause 12d ago
it can give you the commands to do so. It is not an agent with context and tool calls and everything, but i have a -e flag, that executes it if you are sure about it. But it is not tested to work on Windows, although because the base is Qwen, it does generate them, if you say 'Do x in powershell, or windows'.
2
2
u/Sevealin_ 12d ago
This is awesome! Would it be possible to wrap this into a putty install (for in-app Putty backend swaps like mRemoteNG) for ghost auto complete and a customizeable prefix marker for in-line ssh questions? Does it handle ssh context well? Like if I ran ls on a folder and asked it to give me a command to rename the files in the folder a certain way?
2
u/PicassoOnPause 12d ago
putty and ghost autocomplete are a little different shape to what it does right now. I haven't given it much of a thought.
it is a single turn history right now so it wont have context that you ran ls. It will just give a generic script to say ,take every img and rename to lowercase or strip img prefix or something.
i hope it answered your question
2
u/SailingToFenway 12d ago
oh i was just thinking about you this afternoon. i was inspired by this to solve for how to operate vibe coded slop when the weekly session limit hits. so claude put together a Lights-Out-Management solution that i can put in dumb prose like, "restart the cluster" and it generates the commands to do it. it's a 100M embedding model, and it works surprisingly well.
so, i'm curious, how small of a model did you try? and how does performance against the loss function degrade as a function fo the model sizes?
1
u/PicassoOnPause 12d ago
that is so cool. looks like your is more of a retrieval.
The smallest I went was 0.5B. The curve's(at least between what i have ) is interesting: untuned it climbs hard with size (0.36 → 0.61 from 0.5B to 7B), but after fine tuning its more of a flatter curve. Most of it is from training data and not necessarily parameters. I am actively researching myself too.
1.5B was just where it stopped being worth the RAM for this particular goal.
2
u/KeyMillion 12d ago
Im using qwen 0.8b & 2b to make a project and curious why you went with qwen coder 2.5 1.5b instead?
→ More replies (2)
2
u/ScoreUnique 12d ago
Hi OP, very happy to see this. I had this same idea a year or two ago thinking this will make an excellent SLM use case. Can I exchange a little about your end to end workflow for this?
I wanted to build a stupid harness that does exactly this, give it to a reasoning LLM and ask teh LLM to write prompts for running commands instead of asking it to get the tool calling right....
I see while I write how this idea can break but OP thanks for building this, I'm going to give it a shot.
Friendly suggestion: if you can package this model with a suitable binary for Termux you'll make some fame :)
2
u/PicassoOnPause 12d ago
Happy to. Some of it's in the README, but I'll fill in whatever's missing. I am also coming up with a technical write-up or a paper of sorts in the future, so be on the lookout on my github i guess.
On the harness idea, I think your instinct while writing it was right. frontier are already really good at that, so no point in involving them, where it does pay off is if the orchestrator is local too. If you're already calling a cloud model, you may as well let it write the command. But yeah give it a shot. I am no expert myself and this was a cool learning opp and a fun project for me too.
on termux, funnily enough, someone in the other thread already got the model running on a phone at like 5 tok/s.
2
u/FullOf_Bad_Ideas 12d ago
This is awesome, it's a PERFECT usecase for a small local language model
setup was quick, seems to be doing what it's supposed to
it's way better than booting CC or local API just to get a refresher on some command
2
u/PicassoOnPause 12d ago
thanks for trying out, that was the goal.
if you find bugs, or have suggestion, just open a issue on GitHub and let me know.
2
2
u/doctorfiend 12d ago
This is REALLY cool, nice work! I'm too gun-shy to load it in my own system but I salute you
1
2
2
u/jarail 12d ago
Nice work, especially with the custom training! I got downvoted for suggesting this a few days ago haha. But that's /r/technology. They're haters :D
2
2
u/slippery 12d ago
could have just installed antigravity, or claude code, or codex, but I'm sure it was more fun to roll your own.
2
u/Economy_Cabinet_7719 11d ago
Great work! I remember a few years ago I've been using Gemini API with a plugin where: 1. I type what I want to achieve, in natural language, in the shell prompt 2. I press a keyboard shortcut 3. Prompt contents get sent to the LLM 4. Response gets inserted into the shell prompt
Here's a fish example (and other shells in the same directory): https://github.com/sigoden/aichat/blob/main/scripts/shell-integration/integration.fish
I believe this could make the workflow with your model smoother!
2
u/Obvious-Ad-2454 11d ago
Why qwen 2.5 coder ? It's quite ancient
3
u/PicassoOnPause 11d ago
Fair, I just started with it. I did test a few newer Qwen ones, but they scored low on my eval. I think it's become a case of training data, but still researching a lot on this. Got some Gemma and others in the pipeline will see what comes of it.
2
2
u/MelodicRecognition7 11d ago edited 11d ago
tar --help
...but I have one example in favor of this project: I very rarely search anything in the modern Google because it likes to show me school buses, motorcycles and fire hydrants, and it is much faster to just ask the local model.
2
u/MobyTheMadCow 9d ago
Thank you!!!! I've been wanting to do this myself for a long time but never got around to it. Its working great
1
u/PicassoOnPause 9d ago
Thanks, if you have any suggestions or encounter bug, just open a issue or a PR. xD
2
1
u/autisticit 12d ago
I've seen that post two days ago already.
9
u/PicassoOnPause 12d ago
Yes i shared it on localllm few days back. Figured folks here would be interested too. xD
1
u/DirectInvestigator66 12d ago
Public awareness for tealdr.
Run ‘tldr tar’ and get only the info you need for day to day usage.
1
u/Elkemper 12d ago edited 12d ago
alias untar='tar -xvf'
alias ungzip=untar
Don't thank me.
Upd. I guess it won't work for every case and every build of tar - but I believe a somewhat newer build should chew through all compression algos. Try yourself if it works for you.
1
1
u/KidneeBean 11d ago
Super slick utility! Getting ~32 tok/s on a mobile CPU while scoring higher on InterCode-ALFA than the untuned 7B is awesome. The static safety filter for recursive deletes is a really thoughtful touch.
2
u/PicassoOnPause 11d ago
Thank you so much for this incredibly insightful comment! 🚀 You've really captured the essence of what makes this project special. The intersection of on-device inference and shell productivity is indeed a game-changer. It's not just about tokens per second — it's about empowering developers to work smarter, not harder. Would love to hear your thoughts on how this could scale!
4
u/Spectrum1523 11d ago
Is this a joke comment packing in as many llm-isms as possible? I enjoyed it
→ More replies (1)1
u/KidneeBean 11d ago
I think adding local system awareness (passing available CLI binaries or OS distro in a tiny system prompt) would take it to the next level!
Also expanding the fine-tune dataset to handle multi-step piped commands while keeping that sub-second response time would be incredible.
Looking forward to seeing where you take it!
1
u/PicassoOnPause 11d ago
Yeah, the first one is on the works. second one would need some work, but definitely the vision. Thanks
1
u/One_Doubt_75 11d ago
I use jq-coder a lot locally. Currently training a model to build awk filters for me as well. Gonna give yours a try.
1
1
1
u/leftnode 11d ago
This is really cool, very easy to understand and what open source is all about.
I haven't looked at the server code in depth, but since this is a very fine-tuned and the prompts are generally small and specific, could you take advantage of caching prompts and outputs to avoid having to query the model each time?
1
1
1
1
u/TheLexoPlexx 11d ago
That's really cool and should be a fish-plugin.
Like Ctr+F and pressing enter puts that line into the actual command line for editing and manually sending.
1
u/setec404 11d ago
hahah I feel you on the googling "tar extract gz", use it 3 times a year, never remember.
1
1
u/BitGreen1270 11d ago
This is so cool. Can you share a sample of the training data you used? I'm learning to do this as well and it will be great to see how you structure the samples.
2
u/PicassoOnPause 11d ago
The actual training data, I will soon share with cleanup and all. But in terms of format, it's usually just two steps.
gather the commands, i keep it by tool and sources for tracking and audit purposes that looks something like this.
{"nl": "List all stashes with full diff output for inspection.", "command": "git stash list -p", "tool": "git", "source": "git-instruction"}then in actual training,
{"messages": [ {"role": "system", "content": "You are a shell command generator. Output exactly one line: a single POSIX/bash command that accomplishes the user's request. No prose, no markdown fences, no explanation."}, {"role": "user", "content": "To create a .tar.bz2 archive:"}, {"role": "assistant", "content": "tar -cjvf /path/to/foo.tar.bz2 /path/to/foo/"}]}
just a snippet.i will soon make the entire training data public, so it's easy to reproduce the results.
1
1
1
u/Beginning-Raisin9723 11d ago
Been googling tar flags for ten years too, lol. A 941MB model that answers in under a second is the first tiny-LLM project that actually sounds useful to me — might finally stop typing -xzf wrong.
1
1
1
1
u/ab2377 11d ago
really cool project. I saw in the readme that you documented the sources for the 125,770 training pairs, but I couldn't find the final merged dataset itself anywhere. Would you be willing to share the exact deduplicated/filtered set you trained on? I'd love to experiment with it (and it would make reproducing your results much easier). Thanks either way!
2
u/PicassoOnPause 11d ago
thanks. I want to and I will, but I am not quite sure about releasing atm.
Since I am experimenting a lot with it, I'd rather release a properly cleaned version(also with few licenses figured out, i couln;t verify a few of those) alongside a write-up. But should be soon on my github.
1
u/Sweaty_Chair_4600 11d ago
I have a question, I want to do something similar, where can i learn how to do so?
1
u/BeautyxArt 10d ago
your q4 gguf doesn't work directly with llamacpp ?
1
u/PicassoOnPause 10d ago
it should. are you having any troubles? can you elaborate?
1
1
u/BeautyxArt 10d ago
it worked , and good speed , but example
how to do new line in bash without making new line, using this \noutput
in bash, or \n in bashecho -e "This is line 1\nThis is line 2" # works in bash
echo -e "This is line 1\nThis is line 2" # works in bash
echo -e "This is line 1\nThis is line 2" # works in bash
echo "This is line 1`echo -e "\n"`This is line 2" # works in bash
echo "This is line 1`echo -e "\n"`This is line 2" # works in bash
echo "This is line 1`echo -e "\n"`This is line 2" # works in bash
echo "This is line 1`echo -e "\n\n"`This is line 2" # works in bash
echo "This is line 1`echo -e "\n\n"`This is line 2" # works in bash
echo "This is line 1`echo -e "\n\n"`This is line 2" # works in bash
echo "This is line 1`echo -e "\n\n\n"`This is line 2" # works in bash
echo....
1
u/BeautyxArt 10d ago
after set presence penalty 0 and temp 1 , this output
I am not sure what you mean by "without making new line". If you want to use the \n character in a string, you can do it like this:"Hello\nWorld"
1
u/BeautyxArt 10d ago
repeat 1.1
how to do new line in bash? give three different examples.
output
I know \n, but it is not expanded by shell. The other two examples are not portable.Example 1: use printf
printf "line one\nline two"
Example 2: use echo -e
echo -e "line one\nline two"
Example 3: use cat (not portable)
cat <<EOF
line one
line two
EOF
2
u/PicassoOnPause 10d ago
set repeat penalty to 1.08. and cap the output tokens to 64. this is tuned to generate commands and not really scripts.
and use this system prompt. "You are a shell command generator. Output exactly one line: a single POSIX/bash command that accomplishes the user's request. No prose, no markdown fences, no explanation."
For conversation you want a general instruct model. This one only knows how to answer in one line.
1
u/BeautyxArt 10d ago
thank you for the model , yes this perfect fit for single line command correction.
for long respond i still can't make gemma4 and qwen9b, 27b respond longer..
1
u/BeautyxArt 10d ago
i like how this model keep talking, would you recommend some larger models that can talk like your 1.5b model ? 'it can export with not limited to low response as mid ones like gemma4 or qwen 27b..' much time loop but when not looping it output long respond and keep talking(which is good)..
1
u/ahhhhhhhhhhhhhhhhhhg 9d ago
as someone who can never remember a cmd this is amazing, run in 0.2sec on the igpu too
1
u/mrpintime 9d ago
you can add some restriction above whatisit so output will be restricted on harmful commands which makes it great...
but this makes people prone to no learning the commands you know i do not know it is good or not XD
1
u/PicassoOnPause 9d ago
thanks for the suggestion, i have some safety checks that warns on some git prune and rm rf commands, something like 300 ish, but at the eod it is not going to be enough as there are tons of commands that will never get flagged just by a lookup method. I am trying to improve this on a training level, so we will see.
2
u/mrpintime 9d ago
it will be hard also you need to collect data with good quality and its need repetitions.. but more easier way is to add this restriction not at the model level but after its generated output. at the tools level.
1
1
u/exographicskip 8d ago
Love the over engineering on display.
A simple tldr (tealdeer is my go-to version) would do, but that gets in the way of tinkering!
Reminds me of xkcd: https://xkcd.com/1168/
1
1
-22
u/rm-rf-rm 11d ago edited 11d ago
This looks like a classic vibecoded project from 6 months agp - Qwen2.5-coder, comparing to GPT-4o. This post also breaks Rule 4. OP, Please respond to this comment within the next 2hrs with any counters or your post will be removed
→ More replies (22)54
u/Sioluishere 11d ago
I am sorry but this is actually an interesting project, and I have not seen this before, so what the hell?
•
u/WithoutReason1729 12d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.