Support JRuby on Java >= 9 by yaauie · Pull Request #141 · enkessler/childprocess

yaauie · 2019-01-20T15:36:26Z

The JRuby implementation against Javas <= 8 relied on reflection into a
package-private class java.lang.UNIXProcess to get the pid of a process
running on UNIX/posix systems, but that class was removed in Java 9 which
also adds java.lang.Process#pid().

At load, determine the effective major version of Java, and then define
an implementation that uses modern Java's java.lang.Process#pid() where
available, falling back to the original implementation on legacy Javas.

The JAVA_OPTS environment variable is used by JRuby to launch the JVM with
appropriate options; by adding specific --add-opens flags, we are able to
enable the specific reflective access that is necessary to run in Javas >= 9.

The JRuby implementation against Javas <= 8 relied on reflection into a package-private class `java.lang.UNIXProcess` to get the pid of a process running on UNIX/posix systems, but that class was removed in Java 9 which also adds `java.lang.Process#pid()`. At load, determine the effective major version of Java, and then define an implementation that uses modern Java's `java.lang.Process#pid()` where available, falling back to the original implementation on legacy Javas.

coveralls · 2019-01-20T16:02:00Z

Coverage increased (+0.1%) to 94.825% when pulling c37ed88 on yaauie:support-jruby-on-java-gte-9 into 5c57733 on enkessler:master.

The `JAVA_OPTS` environment variable is used by JRuby to launch the JVM with appropriate options; by adding specific `--add-opens` flags, we are able to enable the specific reflective access that is necessary to run in Javas >= 9. For ci, we set `JAVA_OPTS` with the contents of `JAVA_OPTS_FOR_SPECS` inside the `before_script` in order to avoid invoking all jruby JVMs with the flags.

sds

Looks great, @yaauie. Quick question about your Travis configuration.

sds · 2019-01-22T06:05:46Z

+  include:
+  - rvm: jruby-9.2.5.0
+    jdk: openjdk11
+    env: "JAVA_OPTS_FOR_SPECS='--add-opens java.base/java.io=org.jruby.dist --add-opens java.base/sun.nio.ch=org.jruby.dist'"


I might be missing a nuance of how .travis.yml treats env, but could you remove the before_script and just have the following in your include:

env: - "JAVA_OPTS='--add-opens java.base/java.io=org.jruby.dist --add-opens java.base/sun.nio.ch=org.jruby.dist'"

...or does that not work for some reason? I ask because it would avoid setting JAVA_OPTS="" for all other builds.

Several of the setup commands that run through jruby (and therefore create a JVM) do not need those options, so this was an attempt to only provide them to the invocation that runs rspec. I am open to other options.

this can safely be removed when the childprocess gem supports Java9+ enkessler/childprocess#141

* bump jruby to 9.2 * don't rely on logstash-base docker image * work around webmock ruby 2.5 support * ensure data folder exists in docker * change fixnum and bignum to integer * FileUtils.rmdir to rm_rf this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception if the directory isn't empty. On 2.3 the operation will just not delete the directory silently. * bump jruby to 9.2.5.0 and fix test * make rake default task since prepare pack needs it * Resolve compiler warnings (#10247) There are 3 types of compiler warnings that are either resolved or suppressed: 1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout our codebase to the now "raw" type trigger warnings. In most cases we cannot actually resolve the issue, since the JRuby-provided methods for creating `RubyArray`s still return the raw type, so these have been suppressed. 2. Deprecations: - `RubyString#intern19()` -> `RubyString#intern()` - `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)` - `NativeException`: remove import & reference directly; suppress usage warnings - `RaiseException()`: migrate to equivalent non-deprecated methods wherever possible; in some cases where we are using this in conjunction with the also-deprecated `NativeException` to preserve java stacktraces, there seems to be no non-deprecated path forward, so these cases have been suppressed. 3. Redundant Casts - Resolved * JRuby 9.2 bundler shenanigans (#10266) * Revert "Revert "remove forced dependency on old bundler (#9395)"" This reverts commit bef9841. * plugin management: update internal bundler to 1.17.x APIs * deps: update dev dependency webmock to version compatible with JRuby 9.2 * spec: update Pack fixture to include manticore version that doesn't conflict * build: update gradle to version that has Java 11 support * java11: resolve or suppress deprecation warnings * Remove superfluous flag opting into ParNew GC implementation When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the young generation collector ParNew has been the default since Java 8, making the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and should no longer be specified to work with modern Javas. https://bugs.openjdk.java.net/browse/JDK-8006478 https://openjdk.java.net/jeps/214 * spec: set thread name to example description for easier debugging * spec: prevent errors in testing specs by checking against skip list before using * no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code > This method will, on a best-effort basis, throw a `ConcurrentModificationException` > if it is detected that the mapping function modifies this map during computation. > > -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function) * qa: by default, run integration against Elastic Stack 6.5.x To support development on Logstash on top of Java 11, default to testing against an Elastic Stack that is capable of running on Java 11. * qa: ignore deprecation warnings when comparing offline pack output * qa: add Java 9+ support to ChildProcess dev dependency this can safely be removed when the childprocess gem supports Java9+ enkessler/childprocess#141 * qa: allow connections to localhost in webmock * bump jrjackson version * fix filebeat integration tests * spec: ensure license compliance spec runs first The license compliance spec that validates the licenses of bundled plugins appears to not be compatible with the hooks that we inject into bundler for plugin management, and will fail in obscure ways when run after those hooks have been added. Since those hooks are not necessary for validating licenses, the easiest solution was to ensure that those specs run first, before the VM has been poluted. Since the gradle/junit/rspec bridge that is currently in place runs all specs in the same JVM, we also need to make sure that the rspec "world" is reset before a run, to ensure that it doesn't retain spec definitions from previous runs. Also updates the rake invocation, although I'm not sure it is used any more.

* bump jruby to 9.2 * don't rely on logstash-base docker image * work around webmock ruby 2.5 support * ensure data folder exists in docker * change fixnum and bignum to integer * FileUtils.rmdir to rm_rf this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception if the directory isn't empty. On 2.3 the operation will just not delete the directory silently. * bump jruby to 9.2.5.0 and fix test * make rake default task since prepare pack needs it * Resolve compiler warnings (elastic#10247) There are 3 types of compiler warnings that are either resolved or suppressed: 1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout our codebase to the now "raw" type trigger warnings. In most cases we cannot actually resolve the issue, since the JRuby-provided methods for creating `RubyArray`s still return the raw type, so these have been suppressed. 2. Deprecations: - `RubyString#intern19()` -> `RubyString#intern()` - `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)` - `NativeException`: remove import & reference directly; suppress usage warnings - `RaiseException()`: migrate to equivalent non-deprecated methods wherever possible; in some cases where we are using this in conjunction with the also-deprecated `NativeException` to preserve java stacktraces, there seems to be no non-deprecated path forward, so these cases have been suppressed. 3. Redundant Casts - Resolved * JRuby 9.2 bundler shenanigans (elastic#10266) * Revert "Revert "remove forced dependency on old bundler (elastic#9395)"" This reverts commit bef9841. * plugin management: update internal bundler to 1.17.x APIs * deps: update dev dependency webmock to version compatible with JRuby 9.2 * spec: update Pack fixture to include manticore version that doesn't conflict * build: update gradle to version that has Java 11 support * java11: resolve or suppress deprecation warnings * Remove superfluous flag opting into ParNew GC implementation When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the young generation collector ParNew has been the default since Java 8, making the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and should no longer be specified to work with modern Javas. https://bugs.openjdk.java.net/browse/JDK-8006478 https://openjdk.java.net/jeps/214 * spec: set thread name to example description for easier debugging * spec: prevent errors in testing specs by checking against skip list before using * no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code > This method will, on a best-effort basis, throw a `ConcurrentModificationException` > if it is detected that the mapping function modifies this map during computation. > > -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function) * qa: by default, run integration against Elastic Stack 6.5.x To support development on Logstash on top of Java 11, default to testing against an Elastic Stack that is capable of running on Java 11. * qa: ignore deprecation warnings when comparing offline pack output * qa: add Java 9+ support to ChildProcess dev dependency this can safely be removed when the childprocess gem supports Java9+ enkessler/childprocess#141 * qa: allow connections to localhost in webmock * bump jrjackson version * fix filebeat integration tests * spec: ensure license compliance spec runs first The license compliance spec that validates the licenses of bundled plugins appears to not be compatible with the hooks that we inject into bundler for plugin management, and will fail in obscure ways when run after those hooks have been added. Since those hooks are not necessary for validating licenses, the easiest solution was to ensure that those specs run first, before the VM has been poluted. Since the gradle/junit/rspec bridge that is currently in place runs all specs in the same JVM, we also need to make sure that the rspec "world" is reset before a run, to ensure that it doesn't retain spec definitions from previous runs. Also updates the rake invocation, although I'm not sure it is used any more.

backport of #10279 to 6.x * bump jruby to 9.2 * don't rely on logstash-base docker image * work around webmock ruby 2.5 support * ensure data folder exists in docker * change fixnum and bignum to integer * FileUtils.rmdir to rm_rf this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception if the directory isn't empty. On 2.3 the operation will just not delete the directory silently. * bump jruby to 9.2.5.0 and fix test * make rake default task since prepare pack needs it * Resolve compiler warnings (#10247) There are 3 types of compiler warnings that are either resolved or suppressed: 1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout our codebase to the now "raw" type trigger warnings. In most cases we cannot actually resolve the issue, since the JRuby-provided methods for creating `RubyArray`s still return the raw type, so these have been suppressed. 2. Deprecations: - `RubyString#intern19()` -> `RubyString#intern()` - `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)` - `NativeException`: remove import & reference directly; suppress usage warnings - `RaiseException()`: migrate to equivalent non-deprecated methods wherever possible; in some cases where we are using this in conjunction with the also-deprecated `NativeException` to preserve java stacktraces, there seems to be no non-deprecated path forward, so these cases have been suppressed. 3. Redundant Casts - Resolved * JRuby 9.2 bundler shenanigans (#10266) * Revert "Revert "remove forced dependency on old bundler (#9395)"" This reverts commit bef9841. * plugin management: update internal bundler to 1.17.x APIs * deps: update dev dependency webmock to version compatible with JRuby 9.2 * spec: update Pack fixture to include manticore version that doesn't conflict * build: update gradle to version that has Java 11 support * java11: resolve or suppress deprecation warnings * Remove superfluous flag opting into ParNew GC implementation When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the young generation collector ParNew has been the default since Java 8, making the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and should no longer be specified to work with modern Javas. https://bugs.openjdk.java.net/browse/JDK-8006478 https://openjdk.java.net/jeps/214 * spec: set thread name to example description for easier debugging * spec: prevent errors in testing specs by checking against skip list before using * no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code > This method will, on a best-effort basis, throw a `ConcurrentModificationException` > if it is detected that the mapping function modifies this map during computation. > > -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function) * qa: by default, run integration against Elastic Stack 6.5.x To support development on Logstash on top of Java 11, default to testing against an Elastic Stack that is capable of running on Java 11. * qa: ignore deprecation warnings when comparing offline pack output * qa: add Java 9+ support to ChildProcess dev dependency this can safely be removed when the childprocess gem supports Java9+ enkessler/childprocess#141 * qa: allow connections to localhost in webmock * bump jrjackson version * fix filebeat integration tests * spec: ensure license compliance spec runs first The license compliance spec that validates the licenses of bundled plugins appears to not be compatible with the hooks that we inject into bundler for plugin management, and will fail in obscure ways when run after those hooks have been added. Since those hooks are not necessary for validating licenses, the easiest solution was to ensure that those specs run first, before the VM has been poluted. Since the gradle/junit/rspec bridge that is currently in place runs all specs in the same JVM, we also need to make sure that the rspec "world" is reset before a run, to ensure that it doesn't retain spec definitions from previous runs. Also updates the rake invocation, although I'm not sure it is used any more.

Update ruby-childprocessto 3.0.0. ### Version 3.0.0 / 2019-09-20 * [#156](enkessler/childprocess#156 unused `rubyforge_project` from gemspec * [#160](enkessler/childprocess#160): Remove extension to conditionally install `ffi` gem on Windows platforms * [#160](enkessler/childprocess#160): Remove runtime dependency on `rake` gem ### Version 2.0.0 / 2019-07-11 * [#148](enkessler/childprocess#148): Drop support for Ruby 2.0, 2.1, and 2.2 * [#149](enkessler/childprocess#149): Fix Unix fork reopen to be compatible with Ruby 2.6 * [#152](https://github.com/enkessler/childprocess/pull/152)/[#154](https://github.com/enkessler/childprocess/pull/154): Fix hangs and permission errors introduced in Ruby 2.6 for leader processes of process groups ### Version 1.0.1 / 2019-02-03 * [#143](enkessler/childprocess#144): Fix installs by adding `rake` gem as runtime dependency * [#147](enkessler/childprocess#147): Relax `rake` gem constraint from `< 12` to `< 13` ### Version 1.0.0 / 2019-01-28 * [#134](enkessler/childprocess#134): Add support for non-ASCII characters on Windows * [#132](enkessler/childprocess#132): Install `ffi` gem requirement on Windows only * [#128](enkessler/childprocess#128): Convert environment variable values to strings when `posix_spawn` enabled * [#141](enkessler/childprocess#141): Support JRuby on Java >= 9

Update ruby-childprocessto 3.0.0. ### Version 3.0.0 / 2019-09-20 * [NetBSD#156](enkessler/childprocess#156 unused `rubyforge_project` from gemspec * [NetBSD#160](enkessler/childprocess#160): Remove extension to conditionally install `ffi` gem on Windows platforms * [NetBSD#160](enkessler/childprocess#160): Remove runtime dependency on `rake` gem ### Version 2.0.0 / 2019-07-11 * [NetBSD#148](enkessler/childprocess#148): Drop support for Ruby 2.0, 2.1, and 2.2 * [NetBSD#149](enkessler/childprocess#149): Fix Unix fork reopen to be compatible with Ruby 2.6 * [NetBSD#152](https://github.com/enkessler/childprocess/pull/152)/[#154](https://github.com/enkessler/childprocess/pull/154): Fix hangs and permission errors introduced in Ruby 2.6 for leader processes of process groups ### Version 1.0.1 / 2019-02-03 * [NetBSD#143](enkessler/childprocess#144): Fix installs by adding `rake` gem as runtime dependency * [NetBSD#147](enkessler/childprocess#147): Relax `rake` gem constraint from `< 12` to `< 13` ### Version 1.0.0 / 2019-01-28 * [NetBSD#134](enkessler/childprocess#134): Add support for non-ASCII characters on Windows * [NetBSD#132](enkessler/childprocess#132): Install `ffi` gem requirement on Windows only * [NetBSD#128](enkessler/childprocess#128): Convert environment variable values to strings when `posix_spawn` enabled * [NetBSD#141](enkessler/childprocess#141): Support JRuby on Java >= 9

Update ruby-childprocessto 3.0.0. ### Version 3.0.0 / 2019-09-20 * [#156](enkessler/childprocess#156 unused `rubyforge_project` from gemspec * [#160](enkessler/childprocess#160): Remove extension to conditionally install `ffi` gem on Windows platforms * [#160](enkessler/childprocess#160): Remove runtime dependency on `rake` gem ### Version 2.0.0 / 2019-07-11 * [#148](enkessler/childprocess#148): Drop support for Ruby 2.0, 2.1, and 2.2 * [#149](enkessler/childprocess#149): Fix Unix fork reopen to be compatible with Ruby 2.6 * [#152](https://github.com/enkessler/childprocess/pull/152)/[#154](https://github.com/enkessler/childprocess/pull/154): Fix hangs and permission errors introduced in Ruby 2.6 for leader processes of process groups ### Version 1.0.1 / 2019-02-03 * [#143](enkessler/childprocess#144): Fix installs by adding `rake` gem as runtime dependency * [#147](enkessler/childprocess#147): Relax `rake` gem constraint from `< 12` to `< 13` ### Version 1.0.0 / 2019-01-28 * [#134](enkessler/childprocess#134): Add support for non-ASCII characters on Windows * [#132](enkessler/childprocess#132): Install `ffi` gem requirement on Windows only * [#128](enkessler/childprocess#128): Convert environment variable values to strings when `posix_spawn` enabled * [#141](enkessler/childprocess#141): Support JRuby on Java >= 9

yaauie force-pushed the support-jruby-on-java-gte-9 branch 4 times, most recently from 1c07449 to 893c432 Compare January 20, 2019 16:59

yaauie force-pushed the support-jruby-on-java-gte-9 branch from 893c432 to c37ed88 Compare January 20, 2019 17:13

yaauie mentioned this pull request Jan 22, 2019

Maintainer needed #131

Closed

sds reviewed Jan 22, 2019

View reviewed changes

jsvd pushed a commit to jsvd/logstash that referenced this pull request Jan 22, 2019

qa: add Java 9+ support to ChildProcess dev dependency

59f5b72

this can safely be removed when the childprocess gem supports Java9+ enkessler/childprocess#141

sds merged commit 0e040dd into enkessler:master Jan 23, 2019

sds mentioned this pull request Jan 25, 2019

Cut 1.0.0 release #142

Merged

yaauie added a commit to yaauie/logstash that referenced this pull request Feb 4, 2019

qa: add Java 9+ support to ChildProcess dev dependency

650a7e9

this can safely be removed when the childprocess gem supports Java9+ enkessler/childprocess#141

yaauie deleted the support-jruby-on-java-gte-9 branch June 3, 2020 04:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support JRuby on Java >= 9#141

Support JRuby on Java >= 9#141
sds merged 2 commits into
enkessler:masterfrom
yaauie:support-jruby-on-java-gte-9

yaauie commented Jan 20, 2019

Uh oh!

coveralls commented Jan 20, 2019 •

edited

Loading

Uh oh!

sds left a comment

Uh oh!

sds Jan 22, 2019 •

edited

Loading

Uh oh!

yaauie Jan 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

yaauie commented Jan 20, 2019

Uh oh!

coveralls commented Jan 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sds left a comment

Choose a reason for hiding this comment

Uh oh!

sds Jan 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaauie Jan 22, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coveralls commented Jan 20, 2019 •

edited

Loading

sds Jan 22, 2019 •

edited

Loading