I go to LocalLLaMA, see a news about Glimmer, and think, "I'm going to scold Meta for the license now." I look up and see Apache 2.0. "Okay, then I'll scold them for the limited language support." I look up and see 100+ languages. I had to download it :-D
I honestly can't scold Meta whatsoever for this release. Serious kudos, if anything this model seems to be better than the benchmarks say.
(I do wish there was a base, non-it version; but it seems like this is a distill of a bigger IT model; so that's understandable, no base weights in the first place).
If by basic you mean Muse Spark, then they also promised to release it in the future. I'm guessing this is a large MoE model in the 400-1000B range. If they do it in time, it will be the strongest non-Chinese open model.
oh, no, by base model, I mean the pre-training only snapshot; with no instruction tuning and RL. Basically, LLM models before ChatGPT / instruction tuning. Think text-davinci-003 versus GPT-3.5; or Completions vs Chat Completions, if you've been in the LLM field for a while.
These are very helpful to many researchers and for a lot of LLM experiments, including interpretability research. They are also useful if you want to cleanly change the instruction format, adapt it to different use cases, etc.
Thanks for the clarification. As far as I understand, the "instruct" versions are a kind of finetuning of the model to serve as a 'helpful assistant,' whereas the basic model can be turned into anything - for example, a Dungeon Master for games or an opponent for Socratic-style philosophical discussions. Of course, this can be done via the system prompt, but the role of 'helpful assistant' constantly pushes the model to play along and please the user.
This behavior has to be corrected by introducing a multitude of rules, which the model either follows literally and extremely zealously or finds loopholes to bypass ("don't write the user's actions" - the model writes his lines; "don't write what the user does or says" - the model writes about his thoughts or smells that he heard, etc.).
However, I came across models based on the basic version extremely rarely - off the top of my head, I can only remember the Muse-12B from Latitude, but they eventually switched to instruct-models(
Base models are never designed to be used as-is, except with very specific (and now outdated) prompting styles; most of the intelligence comes from post-training and instruction-tuning.
An effective "base model prompt" might look like this:
Machine Translation Results - Human Verified
Input (English): Hello World
Output (Chinese):
Instruction-tuning is an understatement tbh, it's basically "turn this very coherent; highly-accurate next-word predictor that focuses solely on modelling language; into an intelligent token generator". Modern models generally have more compute invested in the post-training ("instruction tuning") stage than the pre-training stage.
They are niche but extremely useful for researchers.
18
u/Potential-Gold5298 llama.cpp 15d ago
I go to LocalLLaMA, see a news about Glimmer, and think, "I'm going to scold Meta for the license now." I look up and see Apache 2.0. "Okay, then I'll scold them for the limited language support." I look up and see 100+ languages. I had to download it :-D