Update sharded_moe.py to support top2 gate with Tutel

[xenshinu]

Tutel is forced to be unused on k > 1 since #2053
Given the fact that multiple experts per token is very common, and the gather and scatter operation without Tutel is so inefficient, I added support of tutel to top2 gate and tested on pipeline engine. This can be done for any k actually, I'll push that later when I have time to test,

[xenshinu]

@xenshinu please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="University of Michigan"

[xenshinu]

Not sure if this check is needed

    if use_tutel:
        # Tutel doesn't support index values masked with zero
        # so we need to replace masked indices with -1
        indices1_mask = mask1.sum(dim=1) * num_experts - 1
        indices1_s = torch.min(indices1_s, indices1_mask)
        indices2_mask = mask2.sum(dim=1) * num_experts - 1
        indices2_s = torch.min(indices2_s, indices2_mask)

I see that in top1gate,
https://github.com/microsoft/DeepSpeed/blob/66d3d3e94dbdfbbf6535cab66256c238983fc7c3/deepspeed/moe/sharded_moe.py#L252
but when I refer to examples from Tutel
https://github.com/microsoft/Tutel/blob/ab7937bb929bc78111d74261b490da25657a7e5c/tutel/impls/fast_dispatch.py#L143

I didn't see any specify check for non-zero mask.

[loadams]

Hi @xenshinu - FYI you'll need to update from the CLA to DCO requirements in this PR if you're able to?

[xenshinu]

Hi @xenshinu - FYI you'll need to update from the CLA to DCO requirements in this PR if you're able to?

Just reset the code to the previous commit and add my signed-and-off.

@loadams Could you please take care of this PR?

[loadams]

Hi @xenshinu - FYI you'll need to update from the CLA to DCO requirements in this PR if you're able to?

Just reset the code to the previous commit and add my signed-and-off.

@loadams Could you please take care of this PR?

@xenshinu - looks like the formatting check is failing, could you run the pre-commit formatter? pre-commit run --all-files and push those changes?

[loadams]

Hi @xenshinu - FYI you'll need to update from the CLA to DCO requirements in this PR if you're able to?

Just reset the code to the previous commit and add my signed-and-off.
@loadams Could you please take care of this PR?

@xenshinu - looks like the formatting check is failing, could you run the pre-commit formatter? pre-commit run --all-files and push those changes?

@xenshinu - curious if you would be able to follow up on this?

[xenshinu]

Hi @xenshinu - FYI you'll need to update from the CLA to DCO requirements in this PR if you're able to?

Just reset the code to the previous commit and add my signed-and-off.
@loadams Could you please take care of this PR?

@xenshinu - looks like the formatting check is failing, could you run the pre-commit formatter? pre-commit run --all-files and push those changes?

@xenshinu - curious if you would be able to follow up on this?

Thanks for your reminder. I'll push the TopK version soon and finish the format issue.