Java: Extractor succeeds for `codeql test run` despite compilation errors in source file

[Marcono1234]

Version
CodeQL CLI v2.4.6

Description of the issue
(Reported this for github/codeql instead of github/codeql-cli-binaries since this might be specific to the Java extractor.)

It appears the Java extractor executed as part of codeql test run succeeds despite the test .java source files containing compilation errors. When running with --show-extractor-output -vvvv it even logs that it is running with --strict-javac-errors, but still does not fail.

Reproduction steps
Assumes you already have a project with the Java library present; you could for example use https://github.com/github/codeql and perform the following in the test directory of it.

1. Create a test directory with the following Test.java file:

   class Test {
       String test() {
           // Compilation error: missing return
       }
   
       void test2() {
           while (true) { }
   
           // Compilation error: unreachable
           System.out.println("test");
       }
   }

   As indicated by the comments, the file causes compilation errors which can be seen when running javac.
2. Create a Test.ql file (content does not matter much), e.g.:

   import java
   
   from Element e
   where e.fromSource()
   select e

3. Execute the test, either by running codeql test run or by using the VSCode extension
   ❌ The test succeeds (or fails due to missing .expected file), but does not fail due to a Java compilation error

[aschackmull]

This has been fixed recently and should be included in the next CLI version.

[Marcono1234]

Thanks! Looks like this is indeed fixed for CodeQL CLI v2.5.0.