Implement the OLMo architecture

[nopperl]

Implements the recently released open-source OLMo architecture. Tested with allenai/OLMo-1B-hf and allenai/OLMo-7B-hf, should work with allenai/OLMo-1.7-7B and the future OLMo-70B as well. Fixes #5408.

Implementation differences from Llama:

• non-parametric layer norm
• QKV clipping

Test:

./convert-hf-to-gguf.py OLMo-1B-hf --outtype f16
./main -m OLMo-1B-hf/ggml-model-f16.gguf --temp 0.8 -s 1000 -n 50 -p "Language modeling is "

Output:

Language modeling is 
"a process that allows a computer to mimic the way that humans speak 
in order to enable computers to recognize speech in a wide range of natural 
language utterances.  In practice, it involves training a machine to 

Reference (requires transformers>=4.40.0.dev0):

from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
set_seed(1000)
olmo = AutoModelForCausalLM.from_pretrained("allenai/OLMo-1B-hf")
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-1B-hf")
message = ["Language modeling is "]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)
response = olmo.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.8)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])

Output:

Language modeling is 
using the same techniques used for text generation. The goal is to create a semantic representation of the world in terms of a language model. In this lecture we will learn some fundamental modeling techniques and will give examples of semantic representations, both text and images

Note: llm_load_vocab shows mismatch in special tokens definition ( 31/50304 vs 52/50304 ). This is due to special tokens 50254 to 50276, which are sequences of " " of varying length.

[nopperl]

Will upload the GGUF conversions here: https://huggingface.co/collections/nopperl/olmo-gguf-66211a0071b6c3d66303fcf1

[nopperl]

@phymbert thanks for checking, I've removed the superfluous code now. The generation results are not affected.

[nopperl]

As an aside, I confirm that it works with the new allenai/OLMo-1.7-7B-hf as well.

$ main -m OLMo-1.7-7B-hf/ggml-model-f16.gguf --temp 0.8 -s 1000 -n 50 -p "Language modeling is "
...
Language modeling is 
a technique for modeling natural language that has a long history and has been the focus of much recent research.  It is concerned with representing text as a series of words, phrases and sentences.  The words are the basic units, and there is
...

[reneleonhardt]

Will upload the GGUF conversions here: https://huggingface.co/collections/nopperl/olmo-gguf-66211a0071b6c3d66303fcf1

Thank you very much! Do you think that some of them would be useful for coding prompts like in CodeGPT?
https://huggingface.co/nopperl/OLMo-1.7-7B-GGUF/tree/main

[nopperl]

Will upload the GGUF conversions here: https://huggingface.co/collections/nopperl/olmo-gguf-66211a0071b6c3d66303fcf1

Thank you very much! Do you think that some of them would be useful for coding prompts like in CodeGPT? https://huggingface.co/nopperl/OLMo-1.7-7B-GGUF/tree/main

If you want to test its performance, I recommend starting from the largest one (f16). However, I did try it on a few coding prompts and I cannot really recommend it for that. For the same weight class (7B), there are better models like deepseek-ai/deepseek-coder-7b-instruct-v1.5 (for QA) or bigcode/starcoder2-7b (for code completion).

Bear in mind that this is still a base model, so it will perform worse than instruction-tuned models on these tasks.