Kernel hang related to noreturn function attributes

[maleadt]

Although Metal.jl works fine on Julia 1.9 locally, it for some reason fails on CI. Maybe this is related to the juliaecosystem machines running an outdated macOS (12.4, while I'm running 13, but 12.6 was also reported to work fine in #85 (comment)).

[christiangnrd]

I'm on 13.2.1 but I can reproduce the hanging behaviour during tests locally. Except for my particular machine, instead of hanging during gpuarrays/random, it consistently hangs during gpuarrays/broadcasting.

I compared activity monitor behaviour between julia --project test/runstests gpuarrays/broadcasting (1.8.5) and julia +beta --project test/runstests gpuarrays/broadcasting (1.9.0-beta4) and in 1.8, for the julia process, cpu is at 97% the whole time and gpu is at ~20% until the test completes, while in 1.9 beta, the cpu is at 100% for a few seconds at the start while gpu is at ~4% for the julia process, then it very quickly drops to 0% for both and the process never finishes.

Another thing I've noticed is that when I stop the test with ctrl+c, I get Distributed.jl warnings telling me that the process was not removed, and indeed looking at activity monitor, I have 5 Julia processes that shouldn't be there.

[maleadt]

Another thing I've noticed is that when I stop the test with ctrl+c, I get Distributed.jl warnings telling me that the process was not removed, and indeed looking at activity monitor, I have 5 Julia processes that shouldn't be there.

That's just the test suite runner capturing your interrupt and exiting. One way to run tests in isolation is, from the Metal.jl repository, do something like julia --project -e 'using Pkg; Pkg.test(; julia_args=gpuarrays/broadcast), where the positional args passed as julia_args indicate the tests you want to run (try passing --help for more information). Hopefully the backtrace you get then from interrupting the process is more informative? Ideally we'd isolate this down to the single operation that makes Metal.jl hang on 1.9.

[christiangnrd]

I'm having trouble getting the test to run outside of the test suite runner because it comes from GPUArrays and I can't figure out how to call it from Metal.

I also did a bisect of Julia, and it seems like JuliaLang/julia@a12c2f0 is the commit that caused the issue (or at least caused it to surface).

[maleadt]

To run code from the GPUArrays test suite, you can do something like:

julia --project=test -L test/setup.jl
julia> AT=MtlArray
julia> eltypes = [Int16, Int32, Int64, Complex{Int16}, Complex{Int32}, Complex{Int64}, ComplexF16, ComplexF32]

That should set-up the required environment. Note that you also might have to start Julia with --check-bounds=yes, like the test runner does.

[christiangnrd]

I found where the test hangs for my machine.
This line, more specifically, here when it tries to run getindex.(Ref(x), 1) on the gpu array. I looked at the generated llvm and I'll be posting each version in their own comment. The one difference of note between the two is that the llvm code from 1.9.0-beta 4 has store atomic {}* %0, {}** %14 release, align 8 that is missing in the 1.8.5 version.

The code I'm referring to is right above the line that looks like

; └└

[christiangnrd]

Julia 1.8.5 output of @code_llvm f(gpu_in...) where f is x->getindex.(Ref(x), 1), and gpu_in is (Int16[0],):

;  @ none within `#315`
define i16 @"julia_#315_1053"({}* nonnull align 8 dereferenceable(32) %0) #0 {
top:
  %gcframe4 = alloca [3 x {}*], align 16
  %gcframe4.sub = getelementptr inbounds [3 x {}*], [3 x {}*]* %gcframe4, i64 0, i64 0
  %1 = bitcast [3 x {}*]* %gcframe4 to i8*
  call void @llvm.memset.p0i8.i32(i8* noundef nonnull align 16 dereferenceable(24) %1, i8 0, i32 24, i1 false)
  %2 = getelementptr inbounds [3 x {}*], [3 x {}*]* %gcframe4, i64 0, i64 2
  %3 = alloca { { {}*, i64 } }, align 8
  %4 = call {}*** inttoptr (i64 7226199708 to {}*** (i64)*)(i64 260) #4
; ┌ @ refpointer.jl:134 within `Ref`
; │┌ @ refvalue.jl:10 within `RefValue` @ refvalue.jl:8
    %5 = bitcast [3 x {}*]* %gcframe4 to i64*
    store i64 4, i64* %5, align 16
    %6 = load {}**, {}*** %4, align 8
    %7 = getelementptr inbounds [3 x {}*], [3 x {}*]* %gcframe4, i64 0, i64 1
    %8 = bitcast {}** %7 to {}***
    store {}** %6, {}*** %8, align 8
    %9 = bitcast {}*** %4 to {}***
    store {}** %gcframe4.sub, {}*** %9, align 8
    %ptls_field5 = getelementptr inbounds {}**, {}*** %4, i64 2
    %10 = bitcast {}*** %ptls_field5 to i8**
    %ptls_load67 = load i8*, i8** %10, align 8
    %11 = call noalias nonnull {}* @ijl_gc_pool_alloc(i8* %ptls_load67, i32 1392, i32 16) #5
    %12 = bitcast {}* %11 to i64*
    %13 = getelementptr inbounds i64, i64* %12, i64 -1
    store atomic i64 4547519648, i64* %13 unordered, align 8
    %14 = bitcast {}* %11 to {}**
    store {}* %0, {}** %14, align 8                                       ; HERE
; └└
; ┌ @ broadcast.jl:860 within `materialize`
   %.fca.0.0.gep = getelementptr inbounds { { {}*, i64 } }, { { {}*, i64 } }* %3, i64 0, i32 0, i32 0
   store {}* %11, {}** %2, align 16
   store {}* %11, {}** %.fca.0.0.gep, align 8
   %.fca.0.1.gep = getelementptr inbounds { { {}*, i64 } }, { { {}*, i64 } }* %3, i64 0, i32 0, i32 1
   store i64 1, i64* %.fca.0.1.gep, align 8
   %15 = call i16 @j_copy_1055({ { {}*, i64 } }* nocapture readonly %3) #0
   %16 = load {}*, {}** %7, align 8
   %17 = bitcast {}*** %4 to {}**
   store {}* %16, {}** %17, align 8
; └
  ret i16 %15
}

[christiangnrd]

Julia 1.9.0-beta4 output of @code_llvm f(gpu_in...) where f is x->getindex.(Ref(x), 1), and gpu_in is (Int16[0],):

;  @ none within `#315`
define i16 @"julia_#315_1018"({}* noundef nonnull align 8 dereferenceable(32) %0) #0 {
top:
  %gcframe4 = alloca [3 x {}*], align 16
  %gcframe4.sub = getelementptr inbounds [3 x {}*], [3 x {}*]* %gcframe4, i64 0, i64 0
  %1 = bitcast [3 x {}*]* %gcframe4 to i8*
  call void @llvm.memset.p0i8.i32(i8* noundef nonnull align 16 dereferenceable(24) %1, i8 0, i32 24, i1 false)
  %2 = getelementptr inbounds [3 x {}*], [3 x {}*]* %gcframe4, i64 0, i64 2
  %3 = alloca { { {}*, i64 } }, align 8
  %4 = call {}*** inttoptr (i64 7226199708 to {}*** (i64)*)(i64 261) #3
; ┌ @ refpointer.jl:136 within `Ref`
; │┌ @ refvalue.jl:10 within `RefValue` @ refvalue.jl:8
    %5 = bitcast [3 x {}*]* %gcframe4 to i64*
    store i64 4, i64* %5, align 16
    %6 = load {}**, {}*** %4, align 8
    %7 = getelementptr inbounds [3 x {}*], [3 x {}*]* %gcframe4, i64 0, i64 1
    %8 = bitcast {}** %7 to {}***
    store {}** %6, {}*** %8, align 8
    %9 = bitcast {}*** %4 to {}***
    store {}** %gcframe4.sub, {}*** %9, align 8
    %ptls_field5 = getelementptr inbounds {}**, {}*** %4, i64 2
    %10 = bitcast {}*** %ptls_field5 to i8**
    %ptls_load67 = load i8*, i8** %10, align 8
    %11 = call noalias nonnull {}* @ijl_gc_pool_alloc(i8* %ptls_load67, i32 1392, i32 16) #4
    %12 = bitcast {}* %11 to i64*
    %13 = getelementptr inbounds i64, i64* %12, i64 -1
    store atomic i64 6245911824, i64* %13 unordered, align 8
    %14 = bitcast {}* %11 to {}**
    store {}* null, {}** %14, align 8                                     ; HERE
    store atomic {}* %0, {}** %14 release, align 8                        ; AND ALSO HERE
; └└
; ┌ @ broadcast.jl:873 within `materialize`
   %.fca.0.0.gep = getelementptr inbounds { { {}*, i64 } }, { { {}*, i64 } }* %3, i64 0, i32 0, i32 0
   store {}* %11, {}** %2, align 16
   store {}* %11, {}** %.fca.0.0.gep, align 8
   %.fca.0.1.gep = getelementptr inbounds { { {}*, i64 } }, { { {}*, i64 } }* %3, i64 0, i32 0, i32 1
   store i64 1, i64* %.fca.0.1.gep, align 8
   %15 = call i16 @j_copy_1020({ { {}*, i64 } }* nocapture readonly %3) #0
   %16 = load {}*, {}** %7, align 8
   %17 = bitcast {}*** %4 to {}**
   store {}* %16, {}** %17, align 8
; └
  ret i16 %15
}

[christiangnrd]

I was hoping the removal of libcmt would magically fix the 1.9 ci issues but unfortunately the tests still hang.

[maleadt]

Sadly this looks codegen related; the atomic store is probably a good clue. I hope I'll have time to investigate next week, but it's problematic that I can't reproduce this.

Which hardware do you have exactly?

[christiangnrd]

I have a 30-core M2 Max Macbook Pro and all of my comments have been using that computer.

I also have access to a base model M1 Mac mini in my lab where I was able to reproduce the hang.

[christiangnrd]

'I ran test Metal many times today on the lab M1 and I have some gist of test outputs. All were run using the currently released version so I don’t know how useful they’ll be, but I got some really weird results on 1.8.5 and 1.9.0-rc1.

Of the three 1.8.5 tests I ran and saved, the first one had a failure in gpuarrays/reductions/== isequal (and gpuarrays/math/power had no tests??), but the other 2 I ran right after passed. I tried to reproduce this failure on my M2 Max but all my 1.8.5 runs passed.
Gist for these tests

Then, I ran the Metal 0.2.0 tests on 1.9.0-rc1, and for the first time since seeing this issue, a test pass completed on 1.9.0, although with some errors. Gist

Hopeful, I reran it on 1.9.0-rc1 twice and unfortunately, both times the test hanged, with some errors in previous tests. The second time had a new error in the unified memory example (hint?) Gist 1, Gist 2

All of the gists are from M1 runs. On my M2 Max, all tests consistently pass on 1.8.5, and I never get any errors on 1.9.0-rc1 (other than broadcasting hanging).

I don’t know how useful these gists will be, but I figure since you can’t reproduce, I might as well give you as much as you can. Last time I had a bug like this that was very inconsistent, it ended being that I wasn’t initializing some values but they weren’t caught in debug mode because all the memory gets 0 initialized when running the debugger.

I’ll run the tests a few more times in the background this weekend from the Master branch to see if anything has changed since there’s been quite a few changes.

[maleadt]

I haven't had the time to reproduce (probably only next week), but since you have a system on which the tests consistently hang: can you post the MWE that makes it hang in a clean session, and could you try running with julia -g2 or after setting MTL_DEBUG_LAYER=1 and MTL_SHADER_VALIDATION=1 in your environment before loading Metal.jl?

[christiangnrd]

MWE (running from the Metal folder):

# Only run the failing test
$ julia --project=test -e'using Pkg; Pkg.add(url="https://github.com/christiangnrd/GPUArrays.jl", rev="metal-mwe-hang")'

# 1.8.5: Passes
$ julia --project=test test/runtests.jl 'gpuarrays/broadcasting'

# 1.8.5: Passes
$ julia -g2 --project=test test/runtests.jl 'gpuarrays/broadcasting'

# 1.9.0-rc1: Fails (hang)
$ Julia +beta --project=test test/runtests.jl 'gpuarrays/broadcasting'

# 1.9.0-rc1: Passes
$ Julia +beta -g2 --project=test test/runtests.jl 'gpuarrays/broadcasting'

[maleadt]

That's not really a minimal example, can you reduce it to any of the tests that make the GPU hang? You mentioned x->getindex.(Ref(x), 1) above?

EDIT: OK, I have access to a system on which this hangs as well. I'll try reducing next week.