Enable the newly added OuterPersistent scheduler

[liqiangxl]

Enable the newly added OuterPersistent scheduler. It requires outer reduction tvs without inner reduction tvs.
Its priority is higher than the original Persistent scheduler which can process inner, outer, and inner_outer.

[liqiangxl]

!build

[liqiangxl]

Disable this deprecated scheduler.

[liqiangxl]

!build

[liqiangxl]

!build

[liqiangxl]

!build

[liqiangxl]

!build

[naoyam]

LGTM

[naoyam]

The codegen diff tests seem to have failed due to a build error:

00:01:57 /usr/bin/ld: libnvfuser_codegen.so: undefined reference to `cuTensorMapEncodeTiled'

This seems to be related to the recent TMA PR by Xiang, but I don't know why the error only shows up now. Any idea? @zasdfgbnm, @xwang233

[zasdfgbnm]

@xwang233 Are we using ccache in our build, and would it help to clean all the cache? I used to use ccache and it used to cause weird error like this and that's why I stopped using it anymore. But this experience was 10+ years ago.

[xwang233]

@zasdfgbnm , hey, I guess the cause of those build issues in codegen diff jobs is that, the base image we used to build your PR was from last night where the TMA PR was not merged to main yet. Any new PR today, after the TMA PR got merged, would be built on top of the "old base image", where the old CMake cache may cause weird conflicts with the new PR.

I suggest that we wait until tomorrow when the new base image, built from main, contains the TMA PR. Those cuTensor related dependency issue should be resolved by then. I'll take a look if CI jobs still have this issue tomorrow and I have some rough idea on where could go wrong.

Ccache is used in our test_container build but not in codegen_diff jobs or manylinux_wheel builds. 🙂

[zasdfgbnm]

@zasdfgbnm , hey, I guess the cause of those build issues in codegen diff jobs is that, the base image we used to build your PR was from last night where the TMA PR was not merged to main yet. Any new PR today, after the TMA PR got merged, would be built on top of the "old base image", where the old CMake cache may cause weird conflicts with the new PR.

I suggest that we wait until tomorrow when the new base image, built from main, contains the TMA PR. Those cuTensor related dependency issue should be resolved by then. I'll take a look if CI jobs still have this issue tomorrow and I have some rough idea on where could go wrong.

Ccache is used in our test_container build but not in codegen_diff jobs or manylinux_wheel builds. 🙂

Sounds like a valid theory. Thanks!

[xwang233]

I think the missing dependency issue in compilation is due to too many concurrency, e.g., 255 jobs there, in the build. The concurrency algorithm (MAX_JOBS) might be fine for local build, but doesn't work that well in servers with large memory and multiple containers running simultaneously. I've reduced the building concurrency in CI codegen diff jobs. Hope this can fix the build issue.