issue with softmax_kernel.cu

[zmxdream]

Hi, I took a look at softmax_ kernel.cu, Is the code customized for the sequence length of power 2, such as 128, 256, 512? Doesn't it seem to apply to a sequence length of 50? right?

I also need to make a PR for the DeepSpeed-Example branch to consolidate these change and won't let our bing-bert example crash!

Originally posted by @zmx19951103 in #587 (comment)

[RezaYazdaniAminabadi]

Hi @zmx19951103,

Yes, it is true that the softmax kernels have some limitation on the sequence length that requires to be dividable by 16. However, we pad both the input and mask in the deepspeed transformer API to address the issue for other sequence length as you mentioned. I have also added some different sequence-length in here to test against them.
Please let me know if you have some issues running the transformer kernel with some configuration that you are testing.
Thanks.
Reza

[zmxdream]

thanks for your reply!