Implement rotary embeddings#7

Merged

thomasw21 merged 13 commits intobigscience-workshop:mainfrom

thomasw21:rotary_embeddings

Jul 28, 2021

Commits on Jul 23, 2021

Integrate EleutherAI's version of rotary embeddings + make some small optimisation
Thomas
committed
Add argument parser for position embeddings
Thomas
committed
Making max-absolute-position-embeddings optional
Thomas
committed
Move enum outside model
Thomas
committed
Handle max_seq_len_cached better
Thomas
committed
Fix dtype issue in rotary embeddings
Thomas
committed
Fix tensor size
Thomas
committed
Replace hidden_dim by hidden_size_per_attention_head
Thomas
committed
Change all examples to new format and improve help in argparser
Thomas
committed
Revert back changes, add comparison with position embedding type when checkpointing and replace args.max_position_embeddings with an upper bound on the sequence sizes
Thomas
committed
Revert back changes:
Thomas
committed
Reformat
Thomas
committed
Rm run.sh~ and modify back run.sh
Thomas
committed