Native (java) extension documentation

[p-mongo]

jruby 9.2.0.0
linux

I had an issue filed with bundler (rubygems/bundler#6678 (comment)) regarding bundler not building bson's java extension under jruby when installing from git. The bundler team said this:

This is because bson explicitly does not set extensions for JRuby.

I so far have not able to find any documentation on what I need to specify for the java extension to be built. https://github.com/jruby/jruby-examples/tree/master/extensions/basic has no gemspec and doesn't specify anything in gemfile either.

How do I make bundler build the java extension of bson when installing from git?

[kares]

believe there's really no "official" way (yet) ... we do have smt similar setup with AR-JDBC that works
of course, it assumes to be able to find javac on PATH - and than does manual compilation + jar build
... will try to get you up with loose instructions and if you try maybe you can document in a wiki page

[kares]

0. you gemspec should have extensions << 'Rakefile' but its tricky since you only want that while the git repo is check-out from Bundler and do not want that kept as the gem gets released (to compile on plain old gem install ...)

Gem::Specification.new do |gem|
  # ...
  if ENV['RELEASE'] != 'true' # convention to disable ext building when releasing gem
    gem.extensions << 'Rakefile' # to support auto-building .jar with :git paths
  end
end

1. in Rakefile, the default task needs to be the one that builds (and cps) the .jar in place
   e.g. task :default => :jar
2. do not rely on rake extension tasks being used - such as the compiler tasks - use bare javac
   ideally your rake should load without any dependencies outside stdlib and rake's own stuff
3. you will need to detect jruby's libraries for class-path setup, currently got ugly since Java 9 :

    classpath = []
    [ 'java.class.path', 'sun.boot.class.path' ].map { |key| ENV_JAVA[key] }.each do |v|
      classpath += v.split(File::PATH_SEPARATOR).find_all { |jar| jar =~ /jruby/i } if v
    end
    # Using Java 9+.  Let's try and infer jruby.jar location from rbconfig
    if classpath.empty?; require 'rbconfig'
      libdir = RbConfig::CONFIG['libdir']
      if libdir.start_with? 'classpath:'
        error "Cannot build with jruby-complete"
      end
      classpath << File.join(libdir, 'jruby.jar')
end

4. do javac ideally in an isolated tmp dir:

      javac = "#{javac} -target #{target} -source #{source} #{args.join(' ')}"
      javac << " #{debug ? '-g' : ''}"
      javac << " -cp \"#{classpath}\" -d #{classes_dir} #{source_files.join(' ')}"
      sh(javac) do |ok|
        raise 'could not build .jar extension - compilation failure' unless ok
      end

5. finally jar up and copy the .jar in place where it would be if you build it for release locally

... you will find a sample tasks (with some noise you shouldn't care about) around here

[p-mongo]

Okay... the extension does get built when not installing it from git, can that process be invoked instead somehow?

[headius]

@p-mongo Why do you want to build the extension on install? There's really no need to do this, since it can be built once and used across platforms. Building Java extensions on install is nothing we've really supported, so I'd like to understand the use case.

[p-mongo]

@headius Please see the bundler issue linked in the top comment. Currently the bson-ruby library is not usable when referencing its git repo in a Gemfile because when bundler installs bson-ruby referenced in this way, the java native extension does not get built.

[p-mongo]

To provide additional context on this, if I want to test e.g. Mongoid against bson-ruby master on jruby, I currently have to do the following:

• Clone bson-ruby
• bundle install
• rake compile
• gem build *.gemspec
• gem install *.gem

Now the installed bson gem is the master one (although its version claims it's the most recent release). I can run rake in Mongoid and it will use this master version of bson.

What I would like to do is go through the steps enumerated when I have bson as a git dependency in gemfile pointing to bson master.

[kares]

you can get some inspiration from AR-JDBC doing a rake jar when checkout by Bundler (from a git: ... source). this is a gem with a native extension that has all its compilation (.jar) dependencies, the tricky part is to only trigger this "on-demand" .jar compilation when using a git snapshot - we do not want this happening on a released gem (which is self contained - has its native ext.jar packaged).

[headius]

Another option to consider would be keeping a built bson jar file in the repository, or setting up a second repository that includes such a jar. I understand the frustration here... the Bundler logic for using gems from git expects them all to work like CRuby gems and either build a C extension or build nothing at all.

We have considered making RubyGems actually build the jar file associate with the gem, but it's so far outside the norm for Java libraries to ship source. There's simply no need, and if we made gems work this way every user would have to have a full JDK rather than just a JRE (Java Runtime Environment, which doesn't ship javac).

[p-mongo]

Is inability to install a JRuby extension from git (without having a prebuilt jar) effectively a limitation of JRuby then? I understand the awkwardness of JRE requirement.

If this is indeed a JRuby limitation then documenting it would be quite helpful as users won't be trying to figure out how to do something that is impossible :)

[eregon]

Since Java 11 there is no more JRE/JDK separation in releases. But there might still be in OS packages.
My thinking is if someone wants to use a gem from git which has .java sources there is no way around, they need to install a JDK if they don't already have it.

The second part is there should be an easy way (a few lines) for gems to just compile .java files into a .jar for such a case without all the complexity of https://github.com/jruby/activerecord-jdbc-adapter/blob/v52.5/Rakefile#L174-L319. And that could be in RubyGems or in JRuby itself.

[headius]

We are not opposed to making it possible to build gems directly out of the repository (there's nothing special about JRuby here) but we do not want to encourage people to ship Java sources in released gem files. We are not C, and the standard means of distribution for JVM-based libraries is class files in jars.

[headius]

The arjdbc Rakefile is also probably the worst example of building an extension for JRuby. There's a lot of leftover crud from when it supported multiple rails versions with each release, and the actual extension building could be simplified greatly. Psych or json or any of the other stdlib extension gems are better examples.