Train baseline + rotary embeddings

[TevenLeScao]

The simplest model, with rotary embeddings. Don't necessarily train to 300B tokens to compare.

[thomasw21]

Rotary embeddings is running similar to tr3:

• scripts are here: Training 4: Rotary embeddings bigscience#2
• job_id: 708913
• user: uty16tp
• link to hub repo: https://huggingface.co/bigscience/tr4-1B3-rotary-tensorboard/tensorboard
• link to checkpoints: TODO
• link to logs: https://huggingface.co/bigscience/tr4-1B3-rotary-logs

[thomasw21]

Update: We've trying to fix some instabilities and re-run all experiments with:

• batch size warmup.
• increase learning rate.

cc @TevenLeScao

[thomasw21]

Update: we have strong suspicion that rotary implementation is wrong. Observations:

• it's not beating absolute embeddings despite other experiments showing otherwise
• It's unstable, a lot more than baseline

Our responsible for the implementation: #7
Tensorboard for 1B3 run: https://huggingface.co/bigscience/tr4-1B3-rotary-tensorboard
Model implementation for rotary: https://github.com/EleutherAI/gpt-neox/blob/main/megatron/model/positional_embeddings.py

[thomasw21]

Closing as we suspect that previous failure with rotary embeddings were linked to warmup