Skip to content

Run cufile tests on Thor (Tegra Linux), skip only on Orin#1422

Merged
rwgk merged 1 commit intoNVIDIA:mainfrom
rwgk:orin_thor_distinction
Jan 5, 2026
Merged

Run cufile tests on Thor (Tegra Linux), skip only on Orin#1422
rwgk merged 1 commit intoNVIDIA:mainfrom
rwgk:orin_thor_distinction

Conversation

@rwgk
Copy link
Copy Markdown
Contributor

@rwgk rwgk commented Dec 24, 2025

Description

Currently the cufile tests are skipped on all Tegra Linux systems, Orin or Thor. However, cufile is supported on Thor. This PR enables the cufile tests accordingly.

Note: This PR is a stop-gap, so that we stop skipping the cufile tests on Thor as soon as possible. Ideally cufile should provide an API to indicate if a given platform is supported, but currently that is not available.

@copy-pr-bot
Copy link
Copy Markdown
Contributor

copy-pr-bot Bot commented Dec 24, 2025

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rwgk
Copy link
Copy Markdown
Contributor Author

rwgk commented Dec 24, 2025

/ok to test

@github-actions

This comment has been minimized.

Sign in to view
@rwgk
Copy link
Copy Markdown
Contributor Author

rwgk commented Dec 24, 2025

@sourabgupta3 for visibility

I could test this PR only on a Thor system with a 13.0 driver (I don't want to spend the time figuring out how to flash the Thor device), which generated a number of errors (see below).

However, the new skip condition code is exercised and working correctly.

Tue Dec 23 22:54:41 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.00                 Driver Version: 595.00         CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Thor (nvgpu)                  N/A  |   N/A              N/A |                  N/A |
| N/A   N/A  N/A             N/A  /  N/A  | Not Supported          |     N/A          N/A |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /home/rgrossekunst/wrk/forked/cuda-python/TestVenv/bin/python
cachedir: .pytest_cache
Using --randomly-seed=3686515587
benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/rgrossekunst/wrk/forked/cuda-python/cuda_bindings
configfile: pyproject.toml
plugins: randomly-4.0.1, benchmark-5.2.3
collecting ... INFO: cuFile library available, version: 1160
INFO: cuFile library version: 1160
INFO: Current filesystem type (findmnt): ext4
INFO: cuFile library version: 1160
INFO: cuFile library version: 1160
collected 28 items

tests/test_cufile.py::test_get_bar_size_in_kb INFO: GPU BAR size: 0 KB (0.00 GB)
PASSED
tests/test_cufile.py::test_stats_start_stop PASSED
tests/test_cufile.py::test_batch_io_basic PASSED
tests/test_cufile.py::test_cufile_read_write_host_memory PASSED
tests/test_cufile.py::test_cufile_read_write PASSED
tests/test_cufile.py::test_cufile_async_read_write INFO: Using cuFile config: /home/rgrossekunst/wrk/forked/cuda-python/cuda_bindings/tests/cufile.json
FAILED
tests/test_cufile.py::test_cufile_read_async INFO: Using cuFile config: /home/rgrossekunst/wrk/forked/cuda-python/cuda_bindings/tests/cufile.json
FAILED
tests/test_cufile.py::test_get_stats_l1 FAILED
tests/test_cufile.py::test_buf_register_host_memory FAILED
tests/test_cufile.py::test_buf_register_simple FAILED
tests/test_cufile.py::test_cufile_success_defined PASSED
tests/test_cufile.py::test_get_parameter_min_max_value INFO: POLLTHRESHOLD_SIZE_KB: min=4, max=18014398509481980
PASSED
tests/test_cufile.py::test_set_get_parameter_string PASSED
tests/test_cufile.py::test_set_stats_level INFO: Successfully set and verified stats level 0
INFO: Successfully set and verified stats level 1
INFO: Successfully set and verified stats level 2
INFO: Successfully set and verified stats level 3
INFO: Correctly caught error for invalid stats level: INVALID_VALUE (5022): invalid arguments; CUDA status: CUDA_SUCCESS (0)
INFO: Correctly caught error for invalid stats level: INVALID_VALUE (5022): invalid arguments; CUDA status: CUDA_SUCCESS (0)
PASSED
tests/test_cufile.py::test_batch_io_cancel FAILED
tests/test_cufile.py::test_cufile_write_async INFO: Using cuFile config: /home/rgrossekunst/wrk/forked/cuda-python/cuda_bindings/tests/cufile.json
FAILED
tests/test_cufile.py::test_set_get_parameter_size_t PASSED
tests/test_cufile.py::test_handle_register PASSED
tests/test_cufile.py::test_set_parameter_posix_pool_slab_array PASSED
tests/test_cufile.py::test_buf_register_invalid_flags FAILED
tests/test_cufile.py::test_cufile_read_write_large FAILED
tests/test_cufile.py::test_batch_io_large_operations FAILED
tests/test_cufile.py::test_buf_register_already_registered FAILED
tests/test_cufile.py::test_get_stats_l3 FAILED
tests/test_cufile.py::test_get_stats_l2 FAILED
tests/test_cufile.py::test_buf_register_multiple_buffers FAILED
tests/test_cufile.py::test_buf_register_large_buffer FAILED
tests/test_cufile.py::test_set_get_parameter_bool PASSED

=================================== FAILURES ===================================
_________________________ test_cufile_async_read_write _________________________

    @pytest.mark.skipif(not isSupportedFilesystem(), reason="cuFile handle_register requires ext4 or xfs filesystem")
    @xfail_handle_register
    @pytest.mark.usefixtures("ctx", "cufile_env_json", "driver")
    def test_cufile_async_read_write():
        """Test cuFile asynchronous read and write operations in sequence."""
        # Create test file
        file_path = "test_cufile_async_rw.bin"
        fd = os.open(file_path, os.O_CREAT | os.O_RDWR | os.O_DIRECT, 0o600)
    
        try:
            # Register file handle
            descr = cufile.Descr()
            descr.type = cufile.FileHandleType.OPAQUE_FD
            descr.handle.fd = fd
            descr.fs_ops = 0
            handle = cufile.handle_register(descr.ptr)
    
            # Allocate and register device buffers
            buf_size = 65536  # 64KB, aligned to 4096 bytes (65536 % 4096 == 0)
            err, write_buf = cuda.cuMemAlloc(buf_size)
            assert err == cuda.CUresult.CUDA_SUCCESS
            cufile.buf_register(int(write_buf), buf_size, 0)
    
            err, read_buf = cuda.cuMemAlloc(buf_size)
            assert err == cuda.CUresult.CUDA_SUCCESS
            cufile.buf_register(int(read_buf), buf_size, 0)
    
            # Create CUDA stream
            err, stream = cuda.cuStreamCreate(0)
            assert err == cuda.CUresult.CUDA_SUCCESS
    
            # Register stream with cuFile
            cufile.stream_register(int(stream), 0)
    
            # Prepare test data in write buffer
            test_string = b"Async RW test data for cuFile!"
            test_string_len = len(test_string)
            repetitions = buf_size // test_string_len
            test_data = test_string * repetitions
            test_data = test_data[:buf_size]  # Ensure it fits exactly in buffer
            host_buf = ctypes.create_string_buffer(test_data, buf_size)
            cuda.cuMemcpyHtoDAsync(write_buf, host_buf, buf_size, 0)
            cuda.cuStreamSynchronize(0)
    
            # Create parameter arrays for async write
            write_size_p = ctypes.c_size_t(buf_size)
            write_file_offset_p = ctypes.c_int64(0)
            write_buf_ptr_offset_p = ctypes.c_int64(0)
            bytes_written_p = ctypes.c_ssize_t(0)
    
            # Perform async write
            cufile.write_async(
                int(handle),
                int(write_buf),
                ctypes.addressof(write_size_p),
                ctypes.addressof(write_file_offset_p),
                ctypes.addressof(write_buf_ptr_offset_p),
                ctypes.addressof(bytes_written_p),
                int(stream),
            )
    
            # Synchronize stream to wait for write completion
            cuda.cuStreamSynchronize(stream)
    
            # Verify bytes written
>           assert bytes_written_p.value == buf_size, f"Expected {buf_size} bytes written, got {bytes_written_p.value}"
E           AssertionError: Expected 65536 bytes written, got 0
E           assert 0 == 65536
E            +  where 0 = c_long(0).value

buf_size   = 65536
bytes_written_p = c_long(0)
descr      = <cuda.bindings.cufile.Descr object at 0xffff8a4f19c0>
err        = <CUresult.CUDA_SUCCESS: 0>
fd         = 37
file_path  = 'test_cufile_async_rw.bin'
handle     = -6076574521278595035
host_buf   = <ctypes.c_char_Array_65536 object at 0xffff8a4f6750>
read_buf   = <CUdeviceptr 48339419136>
repetitions = 2184
stream     = <CUstream 0x26dab240>
test_data  = b'Async RW test data for cuFile!Async RW test data for cuFile!Async RW test data for cuFile!Async RW test data for cuF...ync RW test data for cuFile!Async RW test data for cuFile!Async RW test data for cuFile!Async RW test data for cuFile!'
test_string = b'Async RW test data for cuFile!'
test_string_len = 30
write_buf  = <CUdeviceptr 48339353600>
write_buf_ptr_offset_p = c_long(0)
write_file_offset_p = c_long(0)
write_size_p = c_ulong(65536)

tests/test_cufile.py:848: AssertionError
------------------------------ Captured log setup ------------------------------
INFO     root:test_cufile.py:38 Using cuFile config: /home/rgrossekunst/wrk/forked/cuda-python/cuda_bindings/tests/cufile.json
____________________________ test_cufile_read_async ____________________________

    @pytest.mark.skipif(not isSupportedFilesystem(), reason="cuFile handle_register requires ext4 or xfs filesystem")
    @pytest.mark.usefixtures("ctx", "cufile_env_json", "driver")
    @xfail_handle_register
    def test_cufile_read_async():
        """Test cuFile asynchronous read operations."""
        # Create test file
        file_path = "test_cufile_read_async.bin"
    
        # First create and write test data without O_DIRECT
        fd_temp = os.open(file_path, os.O_CREAT | os.O_RDWR, 0o600)
        # Create test data that's aligned to 4096 bytes
        test_string = b"Async read test data for cuFile!"
        test_string_len = len(test_string)
        buf_size = 65536  # 64KB, aligned to 4096 bytes
        repetitions = buf_size // test_string_len
        test_data = test_string * repetitions
        test_data = test_data[:buf_size]  # Ensure exact 64KB
        os.write(fd_temp, test_data)
        os.fsync(fd_temp)
        os.close(fd_temp)
    
        # Now open with O_DIRECT for cuFile operations
        fd = os.open(file_path, os.O_RDWR | os.O_DIRECT)
    
        try:
            # Register file handle
            descr = cufile.Descr()
            descr.type = cufile.FileHandleType.OPAQUE_FD
            descr.handle.fd = fd
            descr.fs_ops = 0
            handle = cufile.handle_register(descr.ptr)
    
            # Allocate and register device buffer
            buf_size = 65536  # 64KB, aligned to 4096 bytes (65536 % 4096 == 0)
            err, buf_ptr = cuda.cuMemAlloc(buf_size)
>           assert err == cuda.CUresult.CUDA_SUCCESS
E           AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E            +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E            +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buf_size   = 65536
descr      = <cuda.bindings.cufile.Descr object at 0xffff8a4f3910>
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
fd         = 37
fd_temp    = 37
file_path  = 'test_cufile_read_async.bin'
handle     = -6076574521278595035
repetitions = 2048
test_data  = b'Async read test data for cuFile!Async read test data for cuFile!Async read test data for cuFile!Async read test data... test data for cuFile!Async read test data for cuFile!Async read test data for cuFile!Async read test data for cuFile!'
test_string = b'Async read test data for cuFile!'
test_string_len = 32

tests/test_cufile.py:727: AssertionError
------------------------------ Captured log setup ------------------------------
INFO     root:test_cufile.py:38 Using cuFile config: /home/rgrossekunst/wrk/forked/cuda-python/cuda_bindings/tests/cufile.json
______________________________ test_get_stats_l1 _______________________________

    @pytest.mark.skipif(
        cufileVersionLessThan(1150), reason="cuFile parameter APIs require cuFile library version 13.0 or later"
    )
    @pytest.mark.skipif(not isSupportedFilesystem(), reason="cuFile handle_register requires ext4 or xfs filesystem")
    @pytest.mark.usefixtures("stats")
    @xfail_handle_register
    def test_get_stats_l1():
        """Test cuFile L1 statistics retrieval with file operations."""
        # Create test file directly with O_DIRECT
        file_path = "test_stats_l1.bin"
        fd = os.open(file_path, os.O_CREAT | os.O_RDWR | os.O_DIRECT, 0o600)
    
        try:
            cufile.set_stats_level(1)  # L1 = basic operation counts
            # Start collecting cuFile statistics
            cufile.stats_start()
    
            # Create and initialize the descriptor
            descr = cufile.Descr()
            descr.type = cufile.FileHandleType.OPAQUE_FD
            descr.handle.fd = fd
            descr.fs_ops = 0
    
            # Register the handle
            handle = cufile.handle_register(descr.ptr)
    
            # Allocate CUDA memory
            buffer_size = 4096  # 4KB, aligned to 4096 bytes
            err, buf_ptr = cuda.cuMemAlloc(buffer_size)
>           assert err == cuda.CUresult.CUDA_SUCCESS
E           AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E            +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E            +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buffer_size = 4096
descr      = <cuda.bindings.cufile.Descr object at 0xffff8a51c880>
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
fd         = 37
file_path  = 'test_stats_l1.bin'
handle     = -6076574521278595035

tests/test_cufile.py:1607: AssertionError
________________________ test_buf_register_host_memory _________________________

    @pytest.mark.usefixtures("driver")
    def test_buf_register_host_memory():
        """Test buffer registration with host memory."""
        # Allocate host memory
        buffer_size = 4096  # 4KB, aligned to 4096 bytes
        err, buf_ptr = cuda.cuMemHostAlloc(buffer_size, 0)
>       assert err == cuda.CUresult.CUDA_SUCCESS
E       AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E        +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E        +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buffer_size = 4096
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>

tests/test_cufile.py:221: AssertionError
___________________________ test_buf_register_simple ___________________________

    @pytest.mark.usefixtures("driver")
    def test_buf_register_simple():
        """Simple test for buffer registration with cuFile."""
        # Allocate CUDA memory
        buffer_size = 4096  # 4KB, aligned to 4096 bytes
        err, buf_ptr = cuda.cuMemAlloc(buffer_size)
>       assert err == cuda.CUresult.CUDA_SUCCESS
E       AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E        +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E        +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buffer_size = 4096
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>

tests/test_cufile.py:199: AssertionError
_____________________________ test_batch_io_cancel _____________________________

    @pytest.mark.skipif(not isSupportedFilesystem(), reason="cuFile handle_register requires ext4 or xfs filesystem")
    @pytest.mark.usefixtures("driver")
    @xfail_handle_register
    def test_batch_io_cancel():
        """Test batch IO cancellation."""
        # Create test file
        file_path = "test_batch_cancel.bin"
    
        # Allocate CUDA memory
        buf_size = 4096  # 4KB, aligned to 4096 bytes
        num_operations = 2
    
        buffers = []
        for i in range(num_operations):
            err, buf = cuda.cuMemAlloc(buf_size)
>           assert err == cuda.CUresult.CUDA_SUCCESS
E           AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E            +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E            +    where <enum 'CUresult'> = cuda.CUresult

buf        = None
buf_size   = 4096
buffers    = []
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
file_path  = 'test_batch_cancel.bin'
i          = 0
num_operations = 2

tests/test_cufile.py:1115: AssertionError
___________________________ test_cufile_write_async ____________________________

    @pytest.mark.skipif(not isSupportedFilesystem(), reason="cuFile handle_register requires ext4 or xfs filesystem")
    @pytest.mark.usefixtures("ctx", "cufile_env_json", "driver")
    @xfail_handle_register
    def test_cufile_write_async():
        """Test cuFile asynchronous write operations."""
        # Create test file
        file_path = "test_cufile_write_async.bin"
        fd = os.open(file_path, os.O_CREAT | os.O_RDWR | os.O_DIRECT, 0o600)
    
        try:
            # Register file handle
            descr = cufile.Descr()
            descr.type = cufile.FileHandleType.OPAQUE_FD
            descr.handle.fd = fd
            descr.fs_ops = 0
            handle = cufile.handle_register(descr.ptr)
    
            # Allocate and register device buffer
            buf_size = 65536  # 64KB, aligned to 4096 bytes (65536 % 4096 == 0)
            err, buf_ptr = cuda.cuMemAlloc(buf_size)
>           assert err == cuda.CUresult.CUDA_SUCCESS
E           AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E            +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E            +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buf_size   = 65536
descr      = <cuda.bindings.cufile.Descr object at 0xfffd7851bf70>
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
fd         = 37
file_path  = 'test_cufile_write_async.bin'
handle     = -6076574521278595035

tests/test_cufile.py:634: AssertionError
------------------------------ Captured log setup ------------------------------
INFO     root:test_cufile.py:38 Using cuFile config: /home/rgrossekunst/wrk/forked/cuda-python/cuda_bindings/tests/cufile.json
_______________________ test_buf_register_invalid_flags ________________________

    @pytest.mark.usefixtures("driver")
    def test_buf_register_invalid_flags():
        """Test buffer registration with invalid flags."""
        # Allocate CUDA memory
        buffer_size = 65536
        err, buf_ptr = cuda.cuMemAlloc(buffer_size)
>       assert err == cuda.CUresult.CUDA_SUCCESS
E       AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E        +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E        +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buffer_size = 65536
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>

tests/test_cufile.py:273: AssertionError
_________________________ test_cufile_read_write_large _________________________

    @pytest.mark.skipif(not isSupportedFilesystem(), reason="cuFile handle_register requires ext4 or xfs filesystem")
    @pytest.mark.usefixtures("driver")
    @xfail_handle_register
    def test_cufile_read_write_large():
        """Test cuFile read and write operations with large data."""
        # Create test file
        file_path = "test_cufile_rw_large.bin"
    
        # Allocate large CUDA memory (1MB, aligned to 4096 bytes)
        write_size = 1024 * 1024  # 1MB, aligned to 4096 bytes (1048576 % 4096 == 0)
        err, write_buf = cuda.cuMemAlloc(write_size)
>       assert err == cuda.CUresult.CUDA_SUCCESS
E       AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E        +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E        +    where <enum 'CUresult'> = cuda.CUresult

err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
file_path  = 'test_cufile_rw_large.bin'
write_buf  = None
write_size = 1048576

tests/test_cufile.py:531: AssertionError
________________________ test_batch_io_large_operations ________________________

    @pytest.mark.skipif(not isSupportedFilesystem(), reason="cuFile handle_register requires ext4 or xfs filesystem")
    @pytest.mark.usefixtures("driver")
    @xfail_handle_register
    def test_batch_io_large_operations():
        """Test batch IO with large buffer operations."""
        # Create test file
        file_path = "test_batch_large.bin"
    
        # Allocate large CUDA memory (1MB, aligned to 4096 bytes)
        buf_size = 1024 * 1024  # 1MB, aligned to 4096 bytes
        num_operations = 2
    
        write_buffers = []
        read_buffers = []
        all_buffers = []  # Initialize all_buffers to avoid UnboundLocalError
    
        for i in range(num_operations):
            err, buf = cuda.cuMemAlloc(buf_size)
>           assert err == cuda.CUresult.CUDA_SUCCESS
E           AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E            +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E            +    where <enum 'CUresult'> = cuda.CUresult

all_buffers = []
buf        = None
buf_size   = 1048576
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
file_path  = 'test_batch_large.bin'
i          = 0
num_operations = 2
read_buffers = []
write_buffers = []

tests/test_cufile.py:1202: AssertionError
_____________________ test_buf_register_already_registered _____________________

    @pytest.mark.usefixtures("driver")
    def test_buf_register_already_registered():
        """Test that registering an already registered buffer fails."""
        # Allocate CUDA memory
        buffer_size = 4096  # 4KB, aligned to 4096 bytes
        err, buf_ptr = cuda.cuMemAlloc(buffer_size)
>       assert err == cuda.CUresult.CUDA_SUCCESS
E       AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E        +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E        +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buffer_size = 4096
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>

tests/test_cufile.py:318: AssertionError
______________________________ test_get_stats_l3 _______________________________

    @pytest.mark.skipif(
        cufileVersionLessThan(1150), reason="cuFile parameter APIs require cuFile library version 13.0 or later"
    )
    @pytest.mark.skipif(not isSupportedFilesystem(), reason="cuFile handle_register requires ext4 or xfs filesystem")
    @pytest.mark.usefixtures("stats")
    @xfail_handle_register
    def test_get_stats_l3():
        """Test cuFile L3 statistics retrieval with file operations."""
        # Create test file directly with O_DIRECT
        file_path = "test_stats_l3.bin"
        fd = os.open(file_path, os.O_CREAT | os.O_RDWR | os.O_DIRECT, 0o600)
    
        try:
            cufile.set_stats_level(3)  # L3 = comprehensive diagnostic data
    
            # Start collecting cuFile statistics
            cufile.stats_start()
    
            # Create and initialize the descriptor
            descr = cufile.Descr()
            descr.type = cufile.FileHandleType.OPAQUE_FD
            descr.handle.fd = fd
            descr.fs_ops = 0
    
            # Register the handle
            handle = cufile.handle_register(descr.ptr)
    
            # Allocate CUDA memory
            buffer_size = 16384  # 16KB for comprehensive stats testing
            err, buf_ptr = cuda.cuMemAlloc(buffer_size)
>           assert err == cuda.CUresult.CUDA_SUCCESS
E           AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E            +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E            +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buffer_size = 16384
descr      = <cuda.bindings.cufile.Descr object at 0xfffd7851bb50>
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
fd         = 37
file_path  = 'test_stats_l3.bin'
handle     = -6076574521278595035

tests/test_cufile.py:1772: AssertionError
______________________________ test_get_stats_l2 _______________________________

    @pytest.mark.skipif(
        cufileVersionLessThan(1150), reason="cuFile parameter APIs require cuFile library version 13.0 or later"
    )
    @pytest.mark.skipif(not isSupportedFilesystem(), reason="cuFile handle_register requires ext4 or xfs filesystem")
    @pytest.mark.usefixtures("stats")
    @xfail_handle_register
    def test_get_stats_l2():
        """Test cuFile L2 statistics retrieval with file operations."""
        # Create test file directly with O_DIRECT
        file_path = "test_stats_l2.bin"
        fd = os.open(file_path, os.O_CREAT | os.O_RDWR | os.O_DIRECT, 0o600)
    
        try:
            cufile.set_stats_level(2)  # L2 = detailed performance metrics
    
            # Start collecting cuFile statistics
            cufile.stats_start()
    
            # Create and initialize the descriptor
            descr = cufile.Descr()
            descr.type = cufile.FileHandleType.OPAQUE_FD
            descr.handle.fd = fd
            descr.fs_ops = 0
    
            # Register the handle
            handle = cufile.handle_register(descr.ptr)
    
            # Allocate CUDA memory
            buffer_size = 8192  # 8KB for more detailed stats
            err, buf_ptr = cuda.cuMemAlloc(buffer_size)
>           assert err == cuda.CUresult.CUDA_SUCCESS
E           AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E            +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E            +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buffer_size = 8192
descr      = <cuda.bindings.cufile.Descr object at 0xfffd7851bf40>
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
fd         = 37
file_path  = 'test_stats_l2.bin'
handle     = -6076574521278595035

tests/test_cufile.py:1688: AssertionError
______________________ test_buf_register_multiple_buffers ______________________

    @pytest.mark.usefixtures("driver")
    def test_buf_register_multiple_buffers():
        """Test registering multiple buffers."""
        # Allocate multiple CUDA buffers
        buffer_sizes = [4096, 16384, 65536]  # All aligned to 4096 bytes
        buffers = []
    
        for size in buffer_sizes:
            err, buf_ptr = cuda.cuMemAlloc(size)
>           assert err == cuda.CUresult.CUDA_SUCCESS
E           AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E            +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E            +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buffer_sizes = [4096, 16384, 65536]
buffers    = []
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>
size       = 4096

tests/test_cufile.py:246: AssertionError
________________________ test_buf_register_large_buffer ________________________

    @pytest.mark.usefixtures("driver")
    def test_buf_register_large_buffer():
        """Test buffer registration with a large buffer."""
        # Allocate large CUDA memory (1MB, aligned to 4096 bytes)
        buffer_size = 1024 * 1024  # 1MB, aligned to 4096 bytes (1048576 % 4096 == 0)
        err, buf_ptr = cuda.cuMemAlloc(buffer_size)
>       assert err == cuda.CUresult.CUDA_SUCCESS
E       AssertionError: assert <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700> == <CUresult.CUDA_SUCCESS: 0>
E        +  where <CUresult.CUDA_SUCCESS: 0> = <enum 'CUresult'>.CUDA_SUCCESS
E        +    where <enum 'CUresult'> = cuda.CUresult

buf_ptr    = None
buffer_size = 1048576
err        = <CUresult.CUDA_ERROR_ILLEGAL_ADDRESS: 700>

tests/test_cufile.py:296: AssertionError
=========================== short test summary info ============================
FAILED tests/test_cufile.py::test_cufile_async_read_write - AssertionError: E...
FAILED tests/test_cufile.py::test_cufile_read_async - AssertionError: assert ...
FAILED tests/test_cufile.py::test_get_stats_l1 - AssertionError: assert <CUre...
FAILED tests/test_cufile.py::test_buf_register_host_memory - AssertionError: ...
FAILED tests/test_cufile.py::test_buf_register_simple - AssertionError: asser...
FAILED tests/test_cufile.py::test_batch_io_cancel - AssertionError: assert <C...
FAILED tests/test_cufile.py::test_cufile_write_async - AssertionError: assert...
FAILED tests/test_cufile.py::test_buf_register_invalid_flags - AssertionError...
FAILED tests/test_cufile.py::test_cufile_read_write_large - AssertionError: a...
FAILED tests/test_cufile.py::test_batch_io_large_operations - AssertionError:...
FAILED tests/test_cufile.py::test_buf_register_already_registered - Assertion...
FAILED tests/test_cufile.py::test_get_stats_l3 - AssertionError: assert <CUre...
FAILED tests/test_cufile.py::test_get_stats_l2 - AssertionError: assert <CUre...
FAILED tests/test_cufile.py::test_buf_register_multiple_buffers - AssertionEr...
FAILED tests/test_cufile.py::test_buf_register_large_buffer - AssertionError:...
======================== 15 failed, 13 passed in 14.00s ========================

@rwgk rwgk marked this pull request as ready for review December 24, 2025 08:24
@copy-pr-bot
Copy link
Copy Markdown
Contributor

copy-pr-bot Bot commented Dec 24, 2025

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

mdboom
mdboom approved these changes Jan 5, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@mdboom mdboom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rwgk
Copy link
Copy Markdown
Contributor Author

rwgk commented Jan 5, 2026

Thanks @mdboom!

@rwgk rwgk merged commit 9e7e7bf into NVIDIA:main Jan 5, 2026
80 checks passed
@rwgk rwgk deleted the orin_thor_distinction branch January 5, 2026 17:37
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jan 5, 2026

Doc Preview CI
Preview removed because the pull request was closed or merged.

@leofang leofang added this to the cuda-python 13-next, 12-next milestone Jan 14, 2026
@leofang leofang modified the milestones: cuda.bindings 13.1.2 & 12.9.6, cuda.bindings next Feb 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mdboom mdboom mdboom approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

cuda.bindings 13.2.0 & 12.9.6

Development

Successfully merging this pull request may close these issues.

3 participants

@rwgk @mdboom @leofang