Affected Version
Noticed in version 0.14.2, looks like it affects version 0.14.0 through to 0.15.1
Description
Upgrading out test clusters to 0.14.2 resulted in Hadoop Ingestion Tasks failing with the below stack trace. It looks like since 0.14.0 the dependency hadoop-mapreduce-client-core-{$version}.jar is no longer present in the extension druid-hdfs-storage.
This dependency is present in the hadoopDependencies folder, however if my understanding of the classloader code is right, we purposefully do not include these jars in the mapreduce job's classpath.
Looks related to #6828 and #6967
java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.jobhistory.EventWriter.<init>(Lorg/apache/hadoop/fs/FSDataOutputStream;)V at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.createEventWriter(JobHistoryEventHandler.java:420) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.setupEventWriter(JobHistoryEventHandler.java:459) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:549) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$1.run(JobHistoryEventHandler.java:318) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Affected Version
Noticed in version 0.14.2, looks like it affects version 0.14.0 through to 0.15.1
Description
Upgrading out test clusters to 0.14.2 resulted in Hadoop Ingestion Tasks failing with the below stack trace. It looks like since 0.14.0 the dependency hadoop-mapreduce-client-core-{$version}.jar is no longer present in the extension druid-hdfs-storage.
This dependency is present in the hadoopDependencies folder, however if my understanding of the classloader code is right, we purposefully do not include these jars in the mapreduce job's classpath.
Looks related to #6828 and #6967
java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.jobhistory.EventWriter.<init>(Lorg/apache/hadoop/fs/FSDataOutputStream;)V at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.createEventWriter(JobHistoryEventHandler.java:420) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.setupEventWriter(JobHistoryEventHandler.java:459) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:549) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$1.run(JobHistoryEventHandler.java:318) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]