[FEATURE]: Can gradient accumulation be used in the pretraining of llama2-70b?

[Pty72]

Describe the feature

Can gradient accumulation be used in the pretraining of llama2-70b?
And if so, how can it be enabled？

[Fridge003]

Hello, we have just supported gradient accumulation on Gemini plugin.
Usage can be referred to docs/source/en/features/gradient_accumulation_with_booster.md, our online tutorials will also be updated within a few days.