Skip to content

[HOTFIX] Changed code to disable k_truss on CUDA 11.4 differently#1813

Merged
ajschmidt8 merged 1 commit intorapidsai:branch-21.08from
rlratzel:branch-21.08-cuda114ktrusshotfix
Sep 16, 2021
Merged

[HOTFIX] Changed code to disable k_truss on CUDA 11.4 differently#1813
ajschmidt8 merged 1 commit intorapidsai:branch-21.08from
rlratzel:branch-21.08-cuda114ktrusshotfix

Conversation

@rlratzel
Copy link
Copy Markdown
Contributor

@rlratzel rlratzel commented Sep 9, 2021

Changed code to disable k_truss on CUDA 11.4 to not use numba.cuda.runtime.get_version() at import time since this creates a CUDA context which breaks dask LocalCUDACluster init (causes a nccl init invalid usage exception).

This code change results in almost the same UX as before, except explicit imports of the k_truss module no longer raise the exception immediately, and instead raise only when a k_truss algo is called.

Tested by ensuring unit tests for the conditional enabling of k_truss still work and also checked that a LocalCUDACluster could properly be created.

See #1811 for reference

@rlratzel rlratzel added bug Something isn't working ! - Hotfix non-breaking Non-breaking change labels Sep 9, 2021
@rlratzel rlratzel self-assigned this Sep 9, 2021
@rlratzel rlratzel requested a review from a team as a code owner September 9, 2021 20:54
@BradReesWork BradReesWork added this to the 21.10 milestone Sep 14, 2021
@BradReesWork
Copy link
Copy Markdown
Member

@raydouglass you need to merge this since I do not have permissions

@ajschmidt8 ajschmidt8 merged commit fa6f0f1 into rapidsai:branch-21.08 Sep 16, 2021
@rlratzel rlratzel deleted the branch-21.08-cuda114ktrusshotfix branch June 17, 2022 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working non-breaking Non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants