SASS/PTX emission behavior

[manopapad]

I just ran some builds passing different CUDA architectures to install.py, here's what comes out (tested on the generated .cu.o file with cuobjdump):

install.py --arch volta produces:

legion .cu files: sass70, ptx70
legate.core .cu files: sass70

install.py --arch "60;70" produces:

legion .cu files: sass60, sass70, ptx70
legate.core .cu files: sass60, ptx60, sass70, ptx70

install.py --arch 60 produces:

legion .cu files: sass60, ptx60
legate.core .cu files: sass60, ptx60

Can we replicate the Legion build behavior in legate.core/cunumeric as well? I.e. include all passed GPU architectures, emit SASS+PTX for latest architecture, just SASS for the rest, and handle named architectures the same way as numbered.

[trxcllnt]

The way we do in the conda build (here) is how to tell CMake "SASS for all archs, SASS+PTX for the latest". This yields:

--generate-code=arch=compute_60,code=[sm_60]
--generate-code=arch=compute_70,code=[sm_70]
--generate-code=arch=compute_75,code=[sm_75]
--generate-code=arch=compute_80,code=[sm_80]
--generate-code=arch=compute_90,code=[compute_90,sm_90]

The --arch volta stuff is custom to legate's CMakeLists.txt. We could export this module in legate's package config in #571 so consumers could call set_cuda_arch_from_names() after finding legate.

[jjwilke]

I am presuming you want to change the behavior only through install.py rather than overriding the behavior of CMAKE_CUDA_ARCHITECTURES? This is easy enough to map in Python to real-XY, virtual-XY to pass through to CMAKE_CUDA_ARCHITECTURES. I would be hesitant to change the behavior of CMAKE_CUDA_ARCHITECURES or use an intermediate Legion_CUDA_ARCH/Legate_CUDA_ARCH variable, though, if you also want magic at the CMake level.

[manopapad]

Actually, it sounds like CUDA_ARCHITECTURES=all does the right thing in the most typical situation. It will presumably build for all real architectures, up to the maximum supported architecture of the currently installed nvcc, and also virtual for the latest. It can't be tuned to end at a specific architecture, only the maximum supported one, but most of the time that's how high we'd want to compile anyway. Then it looks like -DCMAKE_CUDA_ARCHITECTURES=all/--arch all should be all we really need for conda-build. However, when I tried that I noticed that Legion build failed https://gist.github.com/manopapad/1928142c65880b70f2a123756ee19904. @trxcllnt I assume that's why we're not using "all" in conda-build?

[jjwilke]

We could make this code block smarter to recognize the all... but then you'd have to re-construct the "ALL" list explicitly to send to Legion, which we don't want to do and defeats much of the value of the native CMAKE_CUDA_ARCHITECTURES flag.

  if(Legion_USE_CUDA)
    set(Legion_CUDA_ARCH ${CMAKE_CUDA_ARCHITECTURES})
    list(TRANSFORM Legion_CUDA_ARCH REPLACE "-real" "")
    list(TRANSFORM Legion_CUDA_ARCH REPLACE "-virtual" "")
  endif()

This again motivates finally migrating Legion to CUDA as a language... but that was no small undertaking last time I looked.

[manopapad]

I see. Then we'll shelf this discussion until Legion is ready to move to cmake 3.17+. I agree we shouldn't spend significant effort to adapt to Legion's custom handling on this minor issue, especially since there's built-in support that does the right thing, that Legion will presumably move to at some point.

[manopapad]

@trxcllnt I believe this is now fixed, as of the merge of #828?

[trxcllnt]

Yeah should be.