Fix missing jackson jars for hadoop ingestion by jon-wei · Pull Request #8652 · apache/druid

jon-wei · 2019-10-09T00:00:53Z

This PR adds the following jars back to druid-hdfs-storage (pruned in #8230) due to observing issues with hadoop ingestion when they're missing:

jackson-jaxrs-1.9.13.jar
jackson-xc-1.9.13.jar

It also adds an explicit dependency on jackson-core-asl-1.9.13 to indexing-hadoop, this is currently being brought in through the Curator 4.1.0 dependency, but it's needed when using Hadoop/HDFS. Later versions of Curator such as 4.2.0 no longer bring in this dependency, so we shouldn't rely on Curator for that.

2019-10-08T21:04:19,977 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-10-08T21:04:20,463 WARN [main] org.apache.hadoop.fs.FileSystem - Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.web.WebHdfsFileSystem could not be instantiated
2019-10-08T21:04:20,463 WARN [main] org.apache.hadoop.fs.FileSystem - java.lang.NoClassDefFoundError: org/codehaus/jackson/map/ObjectMapper
2019-10-08T21:04:20,463 WARN [main] org.apache.hadoop.fs.FileSystem - java.lang.ClassNotFoundException: org.codehaus.jackson.map.ObjectMapper
2019-10-08T21:04:20,463 WARN [main] org.apache.hadoop.fs.FileSystem - Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.web.SWebHdfsFileSystem could not be instantiated
2019-10-08T21:04:20,464 WARN [main] org.apache.hadoop.fs.FileSystem - java.lang.NoClassDefFoundError: org.apache.hadoop.hdfs.web.WebHdfsFileSystem
Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/jackson/JsonGenerator
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.getDeclaredMethods(Class.java:1975)
	at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:688)
	at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:380)
	at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:399)
	at com.google.inject.internal.BindingBuilder.toInstance(BindingBuilder.java:84)
	at org.apache.druid.storage.hdfs.HdfsStorageDruidModule.configure(HdfsStorageDruidModule.java:122)
	at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
	at com.google.inject.spi.Elements.getElements(Elements.java:110)
	at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198)
	at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
	at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
	at com.google.inject.spi.Elements.getElements(Elements.java:110)
	at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
	at com.google.inject.Guice.createInjector(Guice.java:99)
	at com.google.inject.Guice.createInjector(Guice.java:73)
	at com.google.inject.Guice.createInjector(Guice.java:62)
	at org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:419)
	at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:69)
	at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:56)
	at org.apache.druid.cli.Main.main(Main.java:113)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.JsonGenerator
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 23 more

This PR has:

been self-reviewed.
been tested in a test Druid cluster.

clintropolis

👍 thanks for fixing

gianm · 2019-10-09T00:13:32Z

+      <dependency>
+        <groupId>org.codehaus.jackson</groupId>
+        <artifactId>jackson-jaxrs</artifactId>
+        <version>1.9.13</version>


Please specify versions using dependencyManagement in the central pom. (It makes sure we use the same version everywhere)

Added codehaus.jackson.version in the central pom

gianm · 2019-10-09T00:25:15Z

        <artifactId>hadoop-hdfs-client</artifactId>
        <scope>runtime</scope>
      </dependency>
+      <dependency>


Please include a comment about why these are necessary (they aren't directly used, nor are they pulled in transitively [apparently], which makes them mysterious and in need of a comment).

Added a comment

If the dependency analyze check complains, you can change the scope to runtime for these.

gianm · 2019-10-09T00:25:23Z

            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </dependency>
+        <dependency>


Another comment, please.

Added a comment

gianm

I like this new approach, assuming it works.

gianm · 2019-10-09T02:24:01Z

-              </exclusion>
              <exclusion>
                <groupId>org.codehaus.jackson</groupId>
                <artifactId>jackson-mapper-asl</artifactId>


I think you'll need this one too.

jon-wei · 2019-10-09T03:27:26Z

Updated this to use a more targeted approach the removes the jackson-core-asl and jackson-mapper-asl exclusions from the HDFS storage extension.

gianm

Thanks @jon-wei. Looks nice.

gianm · 2019-10-09T06:54:52Z

The only CI error is the docs build, which I believe was messed up due to #8653. This patch doesn't change docs. So, I'll merge anyway. Thanks @jon-wei!

* Fix missing jackson jars for hadoop ingestion * PR comments * pom ordering * New approach * Remove all jackson-core/mapper-asl exclusions from hdfs storage

Fix missing jackson jars for hadoop ingestion

97c57ea

jon-wei added Bug Area - Batch Ingestion Area - Dependencies labels Oct 9, 2019

clintropolis approved these changes Oct 9, 2019

View reviewed changes

gianm reviewed Oct 9, 2019

View reviewed changes

jon-wei added 3 commits October 8, 2019 17:38

PR comments

af4ff67

pom ordering

0deab68

New approach

2dd9030

gianm approved these changes Oct 9, 2019

View reviewed changes

gianm reviewed Oct 9, 2019

View reviewed changes

Remove all jackson-core/mapper-asl exclusions from hdfs storage

abc8167

gianm approved these changes Oct 9, 2019

View reviewed changes

gianm merged commit 526f04c into apache:master Oct 9, 2019

sixtus pushed a commit to liquidm/druid that referenced this pull request Oct 21, 2019

fixup hadoop classpath issues - see apache#8652

82523d5

clintropolis mentioned this pull request Nov 25, 2019

un-exclude necessary parquet jackson dependencies instead of relying on curator #8939

Merged

jon-wei added this to the 0.17.0 milestone Dec 17, 2019

Conversation

jon-wei commented Oct 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clintropolis left a comment

Choose a reason for hiding this comment

Uh oh!

gianm Oct 9, 2019

Choose a reason for hiding this comment

Uh oh!

jon-wei Oct 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gianm Oct 9, 2019

Choose a reason for hiding this comment

Uh oh!

jon-wei Oct 9, 2019

Choose a reason for hiding this comment

Uh oh!

ccaominh Oct 9, 2019

Choose a reason for hiding this comment

Uh oh!

gianm Oct 9, 2019

Choose a reason for hiding this comment

Uh oh!

jon-wei Oct 9, 2019

Choose a reason for hiding this comment

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

gianm Oct 9, 2019

Choose a reason for hiding this comment

Uh oh!

jon-wei commented Oct 9, 2019

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

gianm commented Oct 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jon-wei commented Oct 9, 2019 •

edited

Loading

jon-wei Oct 9, 2019 •

edited

Loading