Maven artifacts not reliably downloaded

[seanyoung]

Thanks for creating j4rs, it's a very nice crate to use. I am trying to use j4rs in tests, to compare the output of rust library with an existing java library. However, usually this fails in CI, and it looks like the artifacts aren't downloaded: https://github.com/seanyoung/cir/actions/runs/3824139027/jobs/6506015468

Here is a simple reproduction.

use itertools::Itertools;
use j4rs::{Instance, InvocationArg, JavaClass, Jvm, JvmBuilder, MavenArtifact};
use std::collections::HashMap;

fn main() {
    let jvm = JvmBuilder::new().build().unwrap();
    let dbx_artifact = MavenArtifact::from("org.harctoolbox:IrpTransmogrifier:1.2.11");
    jvm.deploy_artifact(&dbx_artifact).unwrap();
    let dbx_artifact = MavenArtifact::from("org.antlr:antlr4:4.9.3");
    jvm.deploy_artifact(&dbx_artifact).unwrap();
    let dbx_artifact = MavenArtifact::from("org.antlr:antlr4-runtime:4.9.3");
    jvm.deploy_artifact(&dbx_artifact).unwrap();

    let irp = jvm.create_instance(
        "java.lang.String",
        &[InvocationArg::try_from("{38.4k,564}<1,-1|1,-3>(16,-8,D:8,S:8,F:8,~F:8,1,^108m,(16,-4,1,^108m)*) [D:0..255,S:0..255=255-D,F:0..255]").unwrap()],
    ).unwrap();

    let protocol = jvm
        .create_instance("org.harctoolbox.irp.Protocol", &[irp.into()])
        .unwrap();

    let param = HashMap::from([("D", 15), ("S", 1), ("F", 0xe4)]);
    let jparam = jvm
        .java_map(JavaClass::String, JavaClass::Long, param)
        .unwrap();

    let res = jvm
        .invoke(&protocol, "toIrSignal", &[jparam.into()])
        .unwrap();

    let frequency: f64 = jvm
        .to_rust(jvm.invoke(&res, "getFrequency", &[]).unwrap())
        .unwrap();
    let duty_cycle: Option<f64> = jvm
        .to_rust(jvm.invoke(&res, "getDutyCycle", &[]).unwrap())
        .unwrap();

    let intro: Vec<u32> = jvm
        .to_rust(jvm.invoke(&res, "getIntroInts", &[]).unwrap())
        .unwrap();

    let repeat: Vec<u32> = jvm
        .to_rust(jvm.invoke(&res, "getRepeatInts", &[]).unwrap())
        .unwrap();

    let ending: Vec<u32> = jvm
        .to_rust(jvm.invoke(&res, "getEndingInts", &[]).unwrap())
        .unwrap();

    println!("carrier:{}", frequency);
    println!("duty_cycle:{:?}", duty_cycle);
    println!("intro:{}", intro.iter().join(","));
    println!("repeat:{}", repeat.iter().join(","));
    println!("ending:{}", ending.iter().join(","));
}

If I run this after cargo clean, it intermittently produces:

$ cargo run
Exception in thread "main" org.astonbitecode.j4rs.errors.InstantiationException: Cannot create instance of org.harctoolbox.irp.Protocol
	at org.astonbitecode.j4rs.api.instantiation.NativeInstantiationImpl.instantiate(NativeInstantiationImpl.java:47)
Caused by: java.lang.ClassNotFoundException: org.harctoolbox.irp.Protocol
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:375)
	at org.astonbitecode.j4rs.utils.Utils.forNameEnhanced(Utils.java:43)
	at org.astonbitecode.j4rs.api.instantiation.NativeInstantiationImpl.createInstance(NativeInstantiationImpl.java:92)
	at org.astonbitecode.j4rs.api.instantiation.NativeInstantiationImpl.instantiate(NativeInstantiationImpl.java:44)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: JavaError("An Exception was thrown by Java... Please check the logs or the console.")', src/main.rs:21:10

If there is anything I can do to help, please let me know.

[astonbitecode]

Hi!

It might not be clear in the README section, but deploying the artifacts should be done before the first Jvm initialization.

The JvmBuilder builds a JVM, and once this is done, the classpath is defined and finalized. j4rs does not support adding libraries to the classpath during runtime.

This is why the jvm.deploy_artifact is better to be invoked in a build script; to ensure the classpath is complete (any jars are downloaded) before the JVM is created by the process (the rust application is executed).

That being said, please note that the deployment does not take care the transitive dependencies yet.

[seanyoung]

It might not be clear in the README section, but deploying the artifacts should be done before the first Jvm initialization.

So how I do this? The JvmBuilder interface has no deploy_artifact function. I get the impression that a Jvm is needed for deploying the artifacts, some of the code is in java, right?

The JvmBuilder builds a JVM, and once this is done, the classpath is defined and finalized. j4rs does not support adding libraries to the classpath during runtime.

So can I create a JVM, deploy the artifacts, drop the JVM and re-create the JVM with the artifacts in the class path?

[astonbitecode]

You may create a Jvm in the build script (with the JvmBuilder) and use this to invoke the deploy_artifact. This will be done during building your rust application (during cargo build execution).

build.rs:

fn main() {
    let jvm = JvmBuilder::new().build().unwrap();
    let dbx_artifact = MavenArtifact::from("org.harctoolbox:IrpTransmogrifier:1.2.11");
    jvm.deploy_artifact(&dbx_artifact).unwrap();
    let dbx_artifact = MavenArtifact::from("org.antlr:antlr4:4.9.3");
    jvm.deploy_artifact(&dbx_artifact).unwrap();
    let dbx_artifact = MavenArtifact::from("org.antlr:antlr4-runtime:4.9.3");
    jvm.deploy_artifact(&dbx_artifact).unwrap();
}

After the build is completed, you are ready to run your executable. In the main function of the executable you will create a Jvm as normal and use this to invoke the create_instance. The classpath at that time will include the org.harctoolbox:IrpTransmogrifier and anything else you deployed during the build script.

main.rs:

fn main() {
    let jvm = JvmBuilder::new().build().unwrap();

    let irp = jvm.create_instance(
        "java.lang.String",
        &[InvocationArg::try_from("{38.4k,564}<1,-1|1,-3>(16,-8,D:8,S:8,F:8,~F:8,1,^108m,(16,-4,1,^108m)*) [D:0..255,S:0..255=255-D,F:0..255]").unwrap()],
    ).unwrap();

    let protocol = jvm
        .create_instance("org.harctoolbox.irp.Protocol", &[irp.into()])
        .unwrap();
}

[seanyoung]

@astonbitecode thank you very much for the detailed explanation.

Out of interest, why does it need to be in build.rs, iow why does this not work:

fn deploy_artifact() {
    let jvm = JvmBuilder::new().build().unwrap();
    let dbx_artifact = MavenArtifact::from("org.harctoolbox:IrpTransmogrifier:1.2.11");
    jvm.deploy_artifact(&dbx_artifact).unwrap();
    let dbx_artifact = MavenArtifact::from("org.antlr:antlr4:4.9.3");
    jvm.deploy_artifact(&dbx_artifact).unwrap();
    let dbx_artifact = MavenArtifact::from("org.antlr:antlr4-runtime:4.9.3");
    jvm.deploy_artifact(&dbx_artifact).unwrap();
}

fn main() {
    deploy_artifact();

    let jvm = JvmBuilder::new().build().unwrap();

    let irp = jvm.create_instance(
        "java.lang.String",
        &[InvocationArg::try_from("{38.4k,564}<1,-1|1,-3>(16,-8,D:8,S:8,F:8,~F:8,1,^108m,(16,-4,1,^108m)*) [D:0..255,S:0..255=255-D,F:0..255]").unwrap()],
    ).unwrap();

    let protocol = jvm
        .create_instance("org.harctoolbox.irp.Protocol", &[irp.into()])
        .unwrap();
}

Thanks again for your help. I would like to avoid having my rust crate depend on a JVM being present, it is only needed for cargo test .

[astonbitecode]

j4rs uses JNI underneath. According to the jni specification:

As of JDK/JRE 1.2, creation of multiple VMs in a single process is not supported.

As you understand, the reason that it works with the build script, is that the build script is executed in a different process and is executed first. When the process of the actual application is executed, j4rs has the jars already available (they are inside a folder called jassets), so it includes them in the classpath.

As for your use case, you may define j4rs as a build dependency.

Cargo.toml:

[build-dependencies]
j4rs = "0.15.0"

This way, the jars can be deployed and then during the tests (having defined j4rs as a dev dependency), the created Jvms will have the classpath correct and your application will not have the j4rs dependency during runtime.

[seanyoung]

Perfect.

@astonbitecode your help is very much appreciated. Thank you 🙏

A few more words in README.md might be helpful, this really was not obvious to me (which is probably my fault!)

[seanyoung]

Would it be possible to download maven artifacts in pure rust in JvmBuilder interface, and avoid this problem?

[seanyoung]

Would it be possible to download maven artifacts in pure rust in JvmBuilder interface, and avoid this problem?

I've created #46 to show how this can be done. It needs some polish but the idea works for me.

[astonbitecode]

There is no doubt that anything implemented with Java code can also be implemented in Rust.

The current artifacts provisioning with Maven is pretty simple. However, back then, I decided it should be done in Java, because the goal is to fully support maven provisioning, including the transitive dependencies. Such task is pretty demanding and complicated and, doing it from scratch, seems like reinventing the wheel. I am thinking to actually use the official maven plugins for it, via something like the mojo-executor. However, I unfortunately did not have the time for it yet.

Even though I totally understand your need to be able to deploy / provision and use the artifacts in one place (the rust executable itself) I would like to propose you to reconsider the current project recommendation. IMO defining dependencies and using dependencies is better to be done in different steps. This is similar to the way that Maven, gradle and even Cargo work. It seems more natural that provisioning is done during building rather during the actual runtime for languages like Rust and Java.

That being said, did you try to impl as proposed above, using the build script and test and you encountered difficulties? I will be happy to help you sorting this out. There are other solutions too, like including the needed jars in your source code and using them during runtime with JvmBuilder.classpath_entry as shown here.

If none of the above fit to your needs, we should sketch out a different solution, using however, the Java implementation (maybe trying stopping the underlying JVM and recreating it, but I am not sure how JNI will behave in that case).

In any case, I would like to thank you for using j4rs and for taking the time to investigate, implement and open the pull request!

[seanyoung]

It would be nice to have better transitive dependency support at some point. The java mojo-executer project you link is unmaintained according to its readme. However, there is a well-maintained rust crate mvn which does seem to fit the bill, although I have not tried it.

Even if the mvn crate does not work, the rust crate pubgrub seems to provide the plumbing needed for version solving.

[astonbitecode]

Thanks for the suggestions, I was not aware of these crates. However, I am still hesitant to use a non-Java solutions for maven handling...

Anyhow, I pushed a fix for this and now, by default, the artifacts being deployed are made available to the JVM during runtime.

The solution includes a custom ClassLoader, that allows adding artifacts during runtime. Java 9 or higher is required to work; this was the only "non-hacky" solution I could come up with.

To allow j4rs be used in lower-than-9 versions too, there is the option JvmBuilder::with_default_classloader. The documentation of the function gives some details.

Please, when you find some time, see if it works ok for you too.

[seanyoung]

Thanks for the suggestions, I was not aware of these crates. However, I am still hesitant to use a non-Java solutions for maven handling...

Yesterday I tried out the mvn crate. It certainly has issues, it does not deal with variable substitution like ${project.version}. There are other maven rust crates to try.

Anyhow, I pushed a fix for this and now, by default, the artifacts being deployed are made available to the JVM during runtime.

Thank you very much, I'll give this a go.

[seanyoung]

As I was just using j4rs for testing purposes, now I'm building the java program during testing rather than fetching it via maven, see seanyoung/cir@19ae506

Thank you for your help, nevertheless.