[CI Problem] [Hexagon] Builder failing due to outdated LLVM

[joshherr-quic]

Branch/PR Failing

#12873

Jenkins Link

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-12873/6/pipeline#step-1503-log-710

Flakiness

After implementing a dense op for float16 on hexagon, there is an instruction selection error in LLVM. It has been patched as of llvm 15.0.1. The builder is running 14 I believe. Ideally, I would like to be able to provide a sha from LLVM main which is guaranteed to have all the fixes required for SotA codegen.

cc @mehrdadh

[driazati]

Unfortunately the process for something like this is a little clunky. The LLVM that runs in CI is part of a Docker image defined in docker/, a separate PR to update that to 15 would just need to change the version for hexagon: https://github.com/apache/tvm/blob/main/docker/install/ubuntu_install_hexagon.sh#L22-L29

Once that's done we'll need to wait for an image build and update the tags in the repo, then #12873 can proceed

[joshherr-quic]

Below are the cmake options required to build an llvm cross compiler for hexagon. Some of the options may be superfluous due to these settings coming from a build that I use locally for other dev work.

-DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD:STRING="Hexagon" -DLLVM_BUILD_RUNTIME:BOOL=ON -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_LIBCXX:BOOL=ON -DLLVM_ENABLE_PROJECTS:STRING="clang;llvm;lld" -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" -DTARGET_TRIPLE:STRING=x86_64-unknown-linux-gnu -DLLVM_DEFAULT_TARGET_TRIPLE:STRING=x86_64-unknown-linux-gnu

[joshherr-quic]

Unfortunately the process for something like this is a little clunky. The LLVM that runs in CI is part of a Docker image defined in docker/, a separate PR to update that to 15 would just need to change the version for hexagon: https://github.com/apache/tvm/blob/main/docker/install/ubuntu_install_hexagon.sh#L22-L29

Once that's done we'll need to wait for an image build and update the tags in the repo, then #12873 can proceed

After looking into this, I was unable to find a suitable pre-built binary on the llvm github releases page. The closest build is for LLVM 15.0.2 on SLES clang+llvm-15.0.2-x86_64-unknown-linux-gnu-sles15.tar.xz

[driazati]

@joshherr-quic is the issue that there is not a binary release for 15.02+ on x86_64? we can always build from source (which we already do in some situations: https://github.com/apache/tvm/blob/main/docker/install/ubuntu1804_manual_install_llvm.sh) if necessary so long as there is an actual release backing it

[joshherr-quic]

@driazati ah I see, wonderful. I'll look into it! Thanks

[joshherr-quic]

Builds with new codegen requirements are building successfully.

[mehrdadh]

@joshherr-quic did you update the docker image on CI?