In the shardformer tests, we did not check the embedding gradients. However, this is extremely important as the embedding is usually the first layer of the module, its correctness kind of ensures the whole backward is correct. Meanwhile, some embedding is a tied weight, it is important to make the tied weight gradient correct.
In the shardformer tests, we did not check the embedding gradients. However, this is extremely important as the embedding is usually the first layer of the module, its correctness kind of ensures the whole backward is correct. Meanwhile, some embedding is a tied weight, it is important to make the tied weight gradient correct.