Should the codegen diff tool only check the kernel itself?

[zasdfgbnm]

Currently, it check the entire generated string, which contains helper functions and the __global__ kernel. I think we should only check the kernel itself. Otherwise, PRs adding new helper functions (such as #995) will fail on all codegen_diff checks (see example pipeline https://github.com/NVIDIA/Fuser/actions/runs/6365268796/job/17282142084).

[zasdfgbnm]

cc: @jacobhinkle @xwang233

[naoyam]

Makes sense to ignore the code that's just copied from the runtime directory. I don't think we should just ignore simply everything above the global function as some parts of the code are not just verbatim copy. I think that's currently only the nvfuser_index_t typedef? That's actually quite important to check.

[jacobhinkle]

There is a PR in the works #996 which diffs the preamble separately from the kernel code, which should address this. It also diffs some stuff like the ptxas -v output to detect changes in registers, cmem, smem etc.

[jacobhinkle]

That PR is almost ready as it has most of the bells and whistles I wanted now. Here is a sample output file (zipped to bypass github's restrictions on uploading HTML files): codediff_f7786819_feda1e1e_binary_tests.html.zip. This was run on #840 which led me to rethink the approach in that PR.

Note that this detects index type changes as well as differences in registers, cmem, smem, gmem, stack size, and spills. Preamble diffs are shown before the kernel diffs, so if preamble changes but no cuda code changes, that is all you will see instead of thousands of trivial diffs.

[naoyam]

Wow, that looks really nice!

[naoyam]

Can you add some more info like CTK and driver versions as well as the actual device name.

[xwang233]

Can you add some more info like CTK and driver versions as well as the actual device name.

Let's make these things optional or at least can be turned off with a flag. We may want to keep those information private for new releases in the future.

[naoyam]

CI results are not publicly visible, right?

[jacobhinkle]

Can you add some more info like CTK and driver versions as well as the actual device name.

I can add that I think. That reminds me I meant to extract the target arch from the ptxas output also.