[Flaky Test] tests/python/unittest/test_target_codegen_vulkan.py::test_vulkan_stress #10779
Description
These tests were found to be flaky (intermittently failing on main or failed in a PR with unrelated changes). As per the docs, these failures will be disabled in a PR that references this issue until the test owners can fix the source of the flakiness.
Test(s)
tests/python/unittest/test_target_codegen_vulkan.py::test_vulkan_stress- segfaulted in run linked below
Details
[2022-03-24T21:22:59.955Z] tests/python/unittest/test_target_codegen_vulkan.py::test_vulkan_stress[vulkan] Fatal Python error: Segmentation fault
[2022-03-24T21:22:59.955Z]
[2022-03-24T21:22:59.955Z] Thread 0x00007fdbf6ffd700 (most recent call first):
[2022-03-24T21:22:59.955Z] File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 256 in __init_handle_by_constructor__
[2022-03-24T21:22:59.955Z] File "/workspace/python/tvm/_ffi/_ctypes/object.py", line 136 in __init_handle_by_constructor__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/target/target.py", line 123 in __init__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/target/target.py", line 255 in check_and_update_host_consist
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/target/target.py", line 249 in check_and_update_host_consist
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 264 in build
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 139 in build_f
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in <listcomp>
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in worker
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 870 in run
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
[2022-03-24T21:22:59.956Z]
[2022-03-24T21:22:59.956Z] Thread 0x00007fdbf77fe700 (most recent call first):
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 233 in __call__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 133 in lower
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 225 in build
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 139 in build_f
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in <listcomp>
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in worker
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 870 in run
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
[2022-03-24T21:22:59.956Z]
[2022-03-24T21:22:59.956Z] Current thread 0x00007fdbf7fff700 (most recent call first):
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 233 in __call__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 133 in lower
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 225 in build
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 139 in build_f
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in <listcomp>
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in worker
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 870 in run
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
[2022-03-24T21:22:59.956Z]
[2022-03-24T21:22:59.956Z] Thread 0x00007fdc20d5a700 (most recent call first):
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 265 in __init_handle_by_constructor__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/_ffi/_ctypes/object.py", line 139 in __init_handle_by_constructor__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 223 in __call__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 278 in build
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 139 in build_f
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in <listcomp>
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in worker
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 870 in run
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
[2022-03-24T21:22:59.956Z]
[2022-03-24T21:22:59.956Z] Thread 0x00007fdc2155b700 (most recent call first):
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/_ffi/_ctypes/object.py", line 114 in __del__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 133 in lower
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 225 in build
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 139 in build_f
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in <listcomp>
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in worker
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 870 in run
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
[2022-03-24T21:22:59.956Z]
[2022-03-24T21:22:59.956Z] Thread 0x00007fdc21d5c700 (most recent call first):
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 233 in __call__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 278 in build
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 139 in build_f
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in <listcomp>
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in worker
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 870 in run
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
[2022-03-24T21:22:59.956Z]
[2022-03-24T21:22:59.956Z] Thread 0x00007fdc3b28e700 (most recent call first):
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 233 in __call__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/te/schedule.py", line 277 in bind
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 138 in build_f
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in <listcomp>
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in worker
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 870 in run
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
[2022-03-24T21:22:59.956Z]
[2022-03-24T21:22:59.956Z] Thread 0x00007fdc77360700 (most recent call first):
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 233 in __call__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 278 in build
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 139 in build_f
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in <listcomp>
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in worker
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 870 in run
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
[2022-03-24T21:22:59.956Z]
[2022-03-24T21:22:59.956Z] Thread 0x00007fdc76b5f700 (most recent call first):
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 233 in __call__
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/runtime/object_generic.py", line 64 in convert_to_object
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/runtime/object_generic.py", line 109 in convert
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/target/target.py", line 239 in check_and_update_host_consist
[2022-03-24T21:22:59.956Z] File "/workspace/python/tvm/driver/build_module.py", line 276 in build
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 139 in build_f
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in <listcomp>
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 143 in worker
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 870 in run
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
[2022-03-24T21:22:59.956Z]
[2022-03-24T21:22:59.956Z] Thread 0x00007fdc7e4be740 (most recent call first):
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 1060 in _wait_for_tstate_lock
[2022-03-24T21:22:59.956Z] File "/usr/lib/python3.7/threading.py", line 1044 in join
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 158 in run_stress
[2022-03-24T21:22:59.956Z] File "/workspace/tests/python/unittest/test_target_codegen_vulkan.py", line 160 in test_vulkan_stress
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/python.py", line 192 in pytest_pyfunc_call
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_callers.py", line 39 in _multicall
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_manager.py", line 80 in _hookexec
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_hooks.py", line 265 in __call__
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/python.py", line 1761 in runtest
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/runner.py", line 166 in pytest_runtest_call
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_callers.py", line 39 in _multicall
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_manager.py", line 80 in _hookexec
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_hooks.py", line 265 in __call__
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/runner.py", line 259 in <lambda>
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/runner.py", line 338 in from_call
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/runner.py", line 259 in call_runtest_hook
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/runner.py", line 219 in call_and_report
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/runner.py", line 130 in runtestprotocol
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_callers.py", line 39 in _multicall
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_manager.py", line 80 in _hookexec
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_hooks.py", line 265 in __call__
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/main.py", line 347 in pytest_runtestloop
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_callers.py", line 39 in _multicall
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_manager.py", line 80 in _hookexec
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_hooks.py", line 265 in __call__
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/main.py", line 322 in _main
[2022-03-24T21:22:59.956Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/main.py", line 268 in wrap_session
[2022-03-24T21:22:59.957Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/main.py", line 315 in pytest_cmdline_main
[2022-03-24T21:22:59.957Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_callers.py", line 39 in _multicall
[2022-03-24T21:22:59.957Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_manager.py", line 80 in _hookexec
[2022-03-24T21:22:59.957Z] File "/usr/local/lib/python3.7/dist-packages/pluggy/_hooks.py", line 265 in __call__
[2022-03-24T21:22:59.957Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/config/__init__.py", line 165 in main
[2022-03-24T21:22:59.957Z] File "/usr/local/lib/python3.7/dist-packages/_pytest/config/__init__.py", line 187 in console_main
[2022-03-24T21:22:59.957Z] File "/usr/local/lib/python3.7/dist-packages/pytest/__main__.py", line 5 in <module>
[2022-03-24T21:22:59.957Z] File "/usr/lib/python3.7/runpy.py", line 85 in _run_code
[2022-03-24T21:22:59.957Z] File "/usr/lib/python3.7/runpy.py", line 193 in _run_module_as_main
Jenkins Links
- https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/2857/pipeline
- https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/2892/pipeline
cc @Lunderberg