From e1f63eafd0060253d21b64382290661d04ce6a93 Mon Sep 17 00:00:00 2001 From: Eric Lunderberg Date: Wed, 8 May 2024 14:43:58 +0000 Subject: [PATCH] [Cuda] Skip FreeDataSpace when CUDA driver is in inconsistent state Prior to this commit, the RAII handler in `NDArray` would always attempt to free a cuda memory allocation on destruction. However, the call to `cudaFree` may throw an exception. If this happens during stack unwinding due to a previously-thrown exception, this causes the program to immediately terminate, making it difficult to identify the source of the original error. This can commonly occur if an async compute kernel performs an illegal memory access. An exception is thrown from the next cuda API call following the asynchronous error, causing the stack to unwind. If the stack contains any `NDArray` instances which reference cuda allocations, the destructor of these `NDArray` instances will attempt to free memory, triggering the segfault. This commit updates the `CUDADeviceAPI::FreeDataSpace` function to check if the program is currently unwinding the stack due to a thrown exception, while the cuda driver has been left in an unrecoverable state. If this occurs, no attempt to free memory is made, as all cuda API calls will result in an error, and the original exception is allowed to propagate. If the cuda driver is in an unrecoverable state, but no exception is currently unwinding the stack, then this may be the first cuda API call to occur after the asynchronous error. In this case, the `cudaFree` call is still performed, which throws the initial exception. --- src/runtime/cuda/cuda_device_api.cc | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/src/runtime/cuda/cuda_device_api.cc b/src/runtime/cuda/cuda_device_api.cc index 1c80397125e4..ae63f9a4b32f 100644 --- a/src/runtime/cuda/cuda_device_api.cc +++ b/src/runtime/cuda/cuda_device_api.cc @@ -142,6 +142,24 @@ class CUDADeviceAPI final : public DeviceAPI { } void FreeDataSpace(Device dev, void* ptr) final { + if (std::uncaught_exceptions() && cudaPeekAtLastError() == cudaErrorIllegalAddress) { + // For most CUDA calls, an error from an API call will be + // immediately reported, and raised as an exception. However, + // errors raised from async kernel execution leave the CUDA + // driver in an inconsistent state. These errors are "sticky", + // and are never cleared. (See [0] for more details.) + // + // If we are currently unwinding the stack due to a thrown + // exception, and the CUDA driver is in an unrecoverable error, + // do not attempt to free the CUDA allocations. Performing any + // CUDA API call while in this state will throw an additional + // exception, causing a segfault. In this case, it is better to + // allow the original error to continue propagating. + // + // [0] https://forums.developer.nvidia.com/t/cuda-errors-determine-sticky-ness/271625 + return; + } + if (dev.device_type == kDLCUDAHost) { VLOG(1) << "freeing host memory"; CUDA_CALL(cudaFreeHost(ptr));