Error messages from `JET.jl` not printed

[kapple19]

Maybe other packages too, like Aqua.jl (haven't tested yet).

[giordano]

Can you please provide a concrete minimum working example? I use ParallelTestRunner.jl extensively with packages which use Aqua (not JET), and the output is printed correctly, so I find the claim very dubious.

[kapple19]

Yeah I should've provided a demonstration in the first place.

I've created https://github.com/kapple19/TempPkg which demonstrates this issue.

Package code containing error: https://github.com/kapple19/TempPkg/blob/d01da436f82ec16cfb1ad274729386c8df2be5e7/src/TempPkg.jl#L1-L9

module TempPkg

public foo

function foo()
    return Bar(2) # Bar is intentionally undefined
end

end # module TempPkg

runtests.jl content: https://github.com/kapple19/TempPkg/blob/d01da436f82ec16cfb1ad274729386c8df2be5e7/test/runtests.jl#L1-L4

using TempPkg
using ParallelTestRunner

runtests(TempPkg, ARGS)

JET call: https://github.com/kapple19/TempPkg/blob/d01da436f82ec16cfb1ad274729386c8df2be5e7/test/jet.jl#L1-L4

using TempPkg
using JET

JET.test_package(TempPkg; target_modules=(TempPkg,))

Output:

(TempPkg) pkg> test
      Status `~/.julia/dev/TempPkg/test/Project.toml`
  [c3a54625] JET v0.11.3
  [d3525ed8] ParallelTestRunner v2.5.0
  [7c2ae950] TempPkg v0.1.0 `.`
  [8dfed614] Test v1.11.0
      Status `~/.julia/dev/TempPkg/Manifest.toml`
  [da1fd8a2] CodeTracking v3.0.0
  [807dbc54] Compiler v0.1.1
  [b5f81e59] IOCapture v1.0.0
  [c3a54625] JET v0.11.3
  [aa1ae85d] JuliaInterpreter v0.10.10
  [70703baa] JuliaSyntax v1.0.2
  [6f1432cf] LoweredCodeUtils v3.5.1
  [1914dd2f] MacroTools v0.5.16
  [36869731] Malt v1.4.0
  [bac558e1] OrderedCollections v1.8.1
  [d3525ed8] ParallelTestRunner v2.5.0
  [aea7be01] PrecompileTools v1.3.3
  [21216c6a] Preferences v1.5.2
  [05181044] RelocatableFolders v1.0.1
  [295af30f] Revise v3.13.2
  [6c6a2e73] Scratch v1.3.0
  [10745b16] Statistics v1.11.1
  [7c2ae950] TempPkg v0.1.0 `.`
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [8ba89e20] Distributed v1.11.0
  [f43a241f] Downloads v1.7.0
  [7b1f6079] FileWatching v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [37e2e46d] LinearAlgebra v1.12.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [ca575930] NetworkOptions v1.3.0
  [44cfe95a] Pkg v1.12.1
  [de0858da] Printf v1.11.0
  [3fa0cd96] REPL v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [6462fe0b] Sockets v1.11.0
  [f489334b] StyledStrings v1.11.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.3.0+1
  [deac9b47] LibCURL_jll v8.15.0+0
  [e37daf67] LibGit2_jll v1.9.0+0
  [29816b5a] LibSSH2_jll v1.11.3+1
  [14a3606d] MozillaCACerts_jll v2025.11.4
  [4536629a] OpenBLAS_jll v0.3.29+0
  [458c3c95] OpenSSL_jll v3.5.4+0
  [83775a58] Zlib_jll v1.3.1+2
  [8e850b90] libblastrampoline_jll v5.15.0+0
  [8e850ede] nghttp2_jll v1.64.0+1
  [3f19e933] p7zip_jll v17.7.0+0
     Testing Running tests...
Running 1 tests using 1 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
               │   Test   │ ──────────────── CPU ──────────────── │
Test  (Worker) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
jet        (1) │     1.55 │   failed at 2026-03-16T07:47:45.723

Test Summary: | Fail  Total  Time
  Overall     |    1      1  5.7s
    jet       |    1      1  1.5s
    FAILURE

Error in testset jet:
Test Failed at /var/home/kapple/.julia/packages/JET/lX05m/src/JETBase.jl:1375
  Expression: (JET.report_package)(TempPkg; toplevel_logger = nothing, target_modules = (TempPkg,))

ERROR: LoadError: Test run finished with errors
in expression starting at /var/home/kapple/.julia/dev/TempPkg/test/runtests.jl:4
ERROR: Package TempPkg errored during testing

But calling the testing code directly provides an error message:

(TempPkg) pkg> activate test
  Activating project at `~/.julia/dev/TempPkg/test`

julia> include("test/jet.jl")
Precompiling TempPkg finished.
  1 dependency successfully precompiled in 1 seconds
JET-test failed at /var/home/kapple/.julia/packages/JET/lX05m/src/JETBase.jl:1375
  Expression: (JET.report_package)(TempPkg; toplevel_logger = nothing, target_modules = (TempPkg,))
  ═════ 1 possible error found ═════
  ┌ foo() @ TempPkg /var/home/kapple/.julia/dev/TempPkg/src/TempPkg.jl:6
  │ `TempPkg.Bar` is not defined: TempPkg.Bar
  └────────────────────
  
ERROR: LoadError: Test.FallbackTestSetException("There was an error during testing")
Stacktrace:
 [1] record(::Test.FallbackTestSet, t::JET.JETTestFailure)
   @ JET ~/.julia/packages/JET/lX05m/src/JETBase.jl:1434
 [2] func_test(func::typeof(JET.report_package), testname::Symbol, args::Any; broken::Bool, skip::Bool, jetconfigs::@Kwargs{…})
   @ JET ~/.julia/packages/JET/lX05m/src/JETBase.jl:1407
 [3] top-level scope
   @ ~/.julia/dev/TempPkg/test/jet.jl:4
in expression starting at /var/home/kapple/.julia/dev/TempPkg/test/jet.jl:4
Some type information was truncated. Use `show(err)` to see complete types.

And just to confirm I'm not crazy:

(TempPkg/test) pkg> activate .
  Activating project at `~/.julia/dev/TempPkg`

(TempPkg) pkg> test
      Status `~/.julia/dev/TempPkg/test/Project.toml`
  [c3a54625] JET v0.11.3
  [d3525ed8] ParallelTestRunner v2.5.0
  [7c2ae950] TempPkg v0.1.0 `.`
  [8dfed614] Test v1.11.0
      Status `~/.julia/dev/TempPkg/Manifest.toml`
  [da1fd8a2] CodeTracking v3.0.0
  [807dbc54] Compiler v0.1.1
  [b5f81e59] IOCapture v1.0.0
  [c3a54625] JET v0.11.3
  [aa1ae85d] JuliaInterpreter v0.10.10
  [70703baa] JuliaSyntax v1.0.2
  [6f1432cf] LoweredCodeUtils v3.5.1
  [1914dd2f] MacroTools v0.5.16
  [36869731] Malt v1.4.0
  [bac558e1] OrderedCollections v1.8.1
  [d3525ed8] ParallelTestRunner v2.5.0
  [aea7be01] PrecompileTools v1.3.3
  [21216c6a] Preferences v1.5.2
  [05181044] RelocatableFolders v1.0.1
  [295af30f] Revise v3.13.2
  [6c6a2e73] Scratch v1.3.0
  [10745b16] Statistics v1.11.1
  [7c2ae950] TempPkg v0.1.0 `.`
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [8ba89e20] Distributed v1.11.0
  [f43a241f] Downloads v1.7.0
  [7b1f6079] FileWatching v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [37e2e46d] LinearAlgebra v1.12.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [ca575930] NetworkOptions v1.3.0
  [44cfe95a] Pkg v1.12.1
  [de0858da] Printf v1.11.0
  [3fa0cd96] REPL v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [6462fe0b] Sockets v1.11.0
  [f489334b] StyledStrings v1.11.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.3.0+1
  [deac9b47] LibCURL_jll v8.15.0+0
  [e37daf67] LibGit2_jll v1.9.0+0
  [29816b5a] LibSSH2_jll v1.11.3+1
  [14a3606d] MozillaCACerts_jll v2025.11.4
  [4536629a] OpenBLAS_jll v0.3.29+0
  [458c3c95] OpenSSL_jll v3.5.4+0
  [83775a58] Zlib_jll v1.3.1+2
  [8e850b90] libblastrampoline_jll v5.15.0+0
  [8e850ede] nghttp2_jll v1.64.0+1
  [3f19e933] p7zip_jll v17.7.0+0
     Testing Running tests...
Running 1 tests using 1 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
               │   Test   │ ──────────────── CPU ──────────────── │
Test  (Worker) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
jet        (1) │     1.55 │   failed at 2026-03-16T07:48:29.204

Test Summary: | Fail  Total  Time
  Overall     |    1      1  5.7s
    jet       |    1      1  1.5s
    FAILURE

Error in testset jet:
Test Failed at /var/home/kapple/.julia/packages/JET/lX05m/src/JETBase.jl:1375
  Expression: (JET.report_package)(TempPkg; toplevel_logger = nothing, target_modules = (TempPkg,))

ERROR: LoadError: Test run finished with errors
in expression starting at /var/home/kapple/.julia/dev/TempPkg/test/runtests.jl:4
ERROR: Package TempPkg errored during testing

It's the following message that is missing when running pkg> test,
but which is present when includeing the test file:

  ═════ 1 possible error found ═════
  ┌ foo() @ TempPkg /var/home/kapple/.julia/dev/TempPkg/src/TempPkg.jl:6
  │ `TempPkg.Bar` is not defined: TempPkg.Bar
  └────────────────────

[giordano]

We capture all the stdout/stderr of the external process

ParallelTestRunner.jl/src/ParallelTestRunner.jl

Lines 261 to 278 in ee40af5

	function stdio_loop(worker::Malt.Worker, io)
	Threads.@spawn while !eof(worker.stdout) && Malt.isrunning(worker)
	try
	bytes = readavailable(worker.stdout)
	write(io, bytes)
	catch
	break
	end
	end
	Threads.@spawn while !eof(worker.stderr) && Malt.isrunning(worker)
	try
	bytes = readavailable(worker.stderr)
	write(io, bytes)
	catch
	break
	end
	end
	end

The question is probably more like: why is JETLS not printing?

[giordano]

The question is probably more like: why is JETLS not printing?

Took me a while, but basically the problem is that the printing of the report only happens when displaying a test report for Test.FallbackTestSet: https://github.com/aviatesk/JET.jl/blob/bcf7cdfc7a422544b15a37d688dc1cef8f36b9e3/src/JETBase.jl#L1428-L1431. I'm mildly sure JET makes assumptions about the testsets you use.

[giordano]

Oh, no, it's because JET relies on Test.TESTSET_PRINT_ENABLE[] for printing:

ParallelTestRunner.jl/src/ParallelTestRunner.jl

Lines 351 to 363 in ee40af5

	@static if VERSION >= v"1.13.0-DEV.1044"
	@with Test.TESTSET_PRINT_ENABLE => false begin
	inner()
	end
	else
	old_print_setting = Test.TESTSET_PRINT_ENABLE[]
	Test.TESTSET_PRINT_ENABLE[] = false
	try
	inner()
	finally
	Test.TESTSET_PRINT_ENABLE[] = old_print_setting
	end
	end

[giordano]

diff --git a/test/jet.jl b/test/jet.jl
index 98eb471..4b076c6 100644
--- a/test/jet.jl
+++ b/test/jet.jl
@@ -1,4 +1,6 @@
 using TempPkg
 using JET
+using Test
 
+Test.TESTSET_PRINT_ENABLE[] = true
 JET.test_package(TempPkg; target_modules=(TempPkg,))

in your package would solve the problem.

[giordano]

@maleadt I believe setting TESTSET_PRINT_ENABLE[] = false comes from JuliaGPU/CUDA.jl#14. I don't know what was the situation back then, but is that setting still relevant nowadays for this package?

[kapple19]

Happy to close this given the workaround.
The true solution is scoped for a different package which warrants opening a PR in that package then?

[kapple19]

I can confirm the workaround works.

With

using TempPkg
using JET

Test.TESTSET_PRINT_ENABLE[] = true

JET.test_package(TempPkg; target_modules=(TempPkg,))

I now have

(TempPkg) pkg> test
      Status `~/.julia/dev/TempPkg/test/Project.toml`
  [c3a54625] JET v0.11.3
  [d3525ed8] ParallelTestRunner v2.5.0
  [7c2ae950] TempPkg v0.1.0 `.`
  [8dfed614] Test v1.11.0
      Status `~/.julia/dev/TempPkg/Manifest.toml`
  [da1fd8a2] CodeTracking v3.0.0
  [807dbc54] Compiler v0.1.1
  [b5f81e59] IOCapture v1.0.0
  [c3a54625] JET v0.11.3
  [aa1ae85d] JuliaInterpreter v0.10.10
  [70703baa] JuliaSyntax v1.0.2
  [6f1432cf] LoweredCodeUtils v3.5.1
  [1914dd2f] MacroTools v0.5.16
  [36869731] Malt v1.4.0
  [bac558e1] OrderedCollections v1.8.1
  [d3525ed8] ParallelTestRunner v2.5.0
  [aea7be01] PrecompileTools v1.3.3
  [21216c6a] Preferences v1.5.2
  [05181044] RelocatableFolders v1.0.1
  [295af30f] Revise v3.13.2
  [6c6a2e73] Scratch v1.3.0
  [10745b16] Statistics v1.11.1
  [7c2ae950] TempPkg v0.1.0 `.`
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [8ba89e20] Distributed v1.11.0
  [f43a241f] Downloads v1.7.0
  [7b1f6079] FileWatching v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [37e2e46d] LinearAlgebra v1.12.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [ca575930] NetworkOptions v1.3.0
  [44cfe95a] Pkg v1.12.1
  [de0858da] Printf v1.11.0
  [3fa0cd96] REPL v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [6462fe0b] Sockets v1.11.0
  [f489334b] StyledStrings v1.11.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.3.0+1
  [deac9b47] LibCURL_jll v8.15.0+0
  [e37daf67] LibGit2_jll v1.9.0+0
  [29816b5a] LibSSH2_jll v1.11.3+1
  [14a3606d] MozillaCACerts_jll v2025.11.4
  [4536629a] OpenBLAS_jll v0.3.29+0
  [458c3c95] OpenSSL_jll v3.5.4+0
  [83775a58] Zlib_jll v1.3.1+2
  [8e850b90] libblastrampoline_jll v5.15.0+0
  [8e850ede] nghttp2_jll v1.64.0+1
  [3f19e933] p7zip_jll v17.7.0+0
     Testing Running tests...
Running 1 tests using 1 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
               │   Test   │ ──────────────── CPU ──────────────── │
Test  (Worker) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
jet        (1) │     4.09 │   failed at 2026-03-17T08:58:26.937

Output generated during execution of 'jet':
┌ jet: JET-test failed at /var/home/kapple/.julia/packages/JET/lX05m/src/JETBase.jl:1375
│   Expression: (JET.report_package)(TempPkg; toplevel_logger = nothing, target_modules = (TempPkg,))
│   ═════ 1 possible error found ═════
│   ┌ foo() @ TempPkg /var/home/kapple/.julia/dev/TempPkg/src/TempPkg.jl:6
│   │ `TempPkg.Bar` is not defined: TempPkg.Bar
│   └────────────────────
└   

Test Summary: | Fail  Total  Time
  Overall     |    1      1  8.6s
    jet       |    1      1  4.1s
    FAILURE

Error in testset jet:
Test Failed at /var/home/kapple/.julia/packages/JET/lX05m/src/JETBase.jl:1375
  Expression: (JET.report_package)(TempPkg; toplevel_logger = nothing, target_modules = (TempPkg,))

ERROR: LoadError: Test run finished with errors
in expression starting at /var/home/kapple/.julia/dev/TempPkg/test/runtests.jl:4
ERROR: Package TempPkg errored during testing

[kapple19]

It might not be due to CUDA.jl because it tries to restore the original value.

https://github.com/JuliaGPU/CUDA.jl/blob/153f819e49a18e9be8726af548a15ee8f69ba750/test/setup.jl#L127-L140:

@static if VERSION >= v"1.13.0-DEV.1044"
    @with Test.TESTSET_PRINT_ENABLE=>false begin
        inner()
    end
else
    old_print_setting = Test.TESTSET_PRINT_ENABLE[]
    Test.TESTSET_PRINT_ENABLE[] = false
    try
        inner()
    finally
        Test.TESTSET_PRINT_ENABLE[] = old_print_setting
    end
end

[giordano]

I mentioned CUDA.jl because this package comes from code which was originally there. The code you linked is exactly the same code I pointed to here above.

During the tests, TESTSET_PRINT_ENABLE is forcibly set to false, it's then restore at the end.

[kapple19]

Ah, my bad.

[maleadt]

I'm not sure why we have that. Let's try getting rid of it?