qmap hangs forever when using do notation

[samuela]

When you accidentally forget the input parameter and write

julia> ThreadPools.qmap(1:10) do
           0
       end
...

It just hangs forever. I would hope it could produce an error instead when the input arguments are incorrect in some way.

[samuela]

In contrast to

julia> ThreadPools.qmap(1:10) do _
           0
       end

which does work as expected.

[samuela]

Also seems to happen when the function signature doesn't exactly match in other ways:

julia> ThreadPools.qmap(1:3) do (a, b)
           0
       end
...

also hangs...

[tro3]

I think this one is Julia's issue with exception handling in threads, but I'll take a look.

[tro3]

Addressed in v1.1.2, which will propagate in the next hour or so. I put an assertion ahead of the threading to handle this.

[danielwe]

I'm on ThreadPools 1.1.3 and Julia 1.5.3 and still have this problem. More precisely, any exception raised within a job that was assigned using dynamic scheduling causes the code to hang forever. When using static scheduling and spawning, exceptions interrupt execution as expected, complete with nested exceptions and stacktraces.

julia> qforeach(_ -> error(), 1:Threads.nthreads())  # does not work
<silence, press Ctrl-C to get back to the prompt>

julia> @qthreads for _ in 1:Threads.nthreads(); error(); end  # does not work
<silence, as above>

julia> Threads.@threads for _ in 1:Threads.nthreads(); error(); end  # works
ERROR: TaskFailedException:

Stacktrace:
 [1] error() at ./error.jl:42
 [2] [...]
Stacktrace:
 [1] wait at ./task.jl:267 [inlined]
 [2] [...]

julia> @bthreads for _ in 1:Threads.nthreads(); error(); end  # works
ERROR: TaskFailedException:

Stacktrace:
 [1] error() at ./error.jl:42
 [2] [...]
Stacktrace:
 [1] wait at ./task.jl:267 [inlined]
 [2] [...]

julia> fetch(@tspawnat 2 error())
ERROR: TaskFailedException:

Stacktrace:
 [1] error() at ./error.jl:42
 [2] [...]
Stacktrace:
 [1] wait at ./task.jl:267 [inlined]
 [2] [...]

[tro3]

Yes, the core "Exception during thread" issue is Julia's. I haven't seen any updates in 1.5 that address this, but I'll query to make sure. If they have added some hooks, I'll fold into ThreadPools. Thx

[danielwe]

Great! Multithreading in Julia is clearly immature and a work in progress, but Threads.@threads and Threads.@spawn actually seem to handle exceptions sensibly on my machine. Not familiar with the lower-level hooks ThreadPools needs, though.

[tro3]

!! Ok, if you are getting decent behavior out of @threads, then I have some work to do to try to match it (the point with Threadpools.@?threads is to match @threads). I'll take a look.

[tro3]

Okay, looks like it is only a @qthreads issue, as @bthreads does now raise the Exception. (I don't think it did in earlier version of Julia, but I could be wrong). I'll dig and see if I can improve exception handling here - it has been a pain for me as well. :-)

[tro3]

Sigh. Nope. This issue is dependent on JuliaLang/julia#32677, which is still pending.

[danielwe]

Thanks for investigating!

[tro3]

Aha! In looking at the Julia issue and wrapping my head around the async scheduling a bit more, I find that this is not dependent on 32677 after all. It is a subtle issue with the QueuePool handler. Will submit an update shortly.