Hi,
I am trying to reproduce the results in Table 2, specifically, the results for "Full-Rank" row in the table for 60m parameter model.
I am running scripts/llm_pretrain/sltrain60m.sh, but I have modified --peft_model from "sltrain" to "full". After 11,000 steps, I get this:
"Eval loss and perplexity at step 11001: 3.4039955139160156, 30.084061520394627"
Whereas the perplexity reported in Table 2 is 34.06
Could you please advise on how to re-produce the Table 2 results? Greatly appreciate your help!
Hi,
I am trying to reproduce the results in Table 2, specifically, the results for "Full-Rank" row in the table for 60m parameter model.
I am running scripts/llm_pretrain/sltrain60m.sh, but I have modified --peft_model from "sltrain" to "full". After 11,000 steps, I get this:
"Eval loss and perplexity at step 11001: 3.4039955139160156, 30.084061520394627"
Whereas the perplexity reported in Table 2 is 34.06
Could you please advise on how to re-produce the Table 2 results? Greatly appreciate your help!