java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HRegionLocation.getServerAddress()Lorg/apache/hadoop/hbase/HServerAddress

[du291]

There appears to be an incompatibility with newer versions of hbase, apparently the method is deprecated and/or removed.

java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HRegionLocation.getServerAddress()Lorg/apache/hadoop/hbase/HServerAddress;
at parallelai.spyglass.hbase.HBaseInputFormatGranular.getSplits(HBaseInputFormatGranular.java:113)
at parallelai.spyglass.hbase.HBaseInputFormatGranular.getSplits(HBaseInputFormatGranular.java:43)
at cascading.tap.hadoop.io.MultiInputFormat.getSplits(MultiInputFormat.java:200)
at cascading.tap.hadoop.io.MultiInputFormat.getSplits(MultiInputFormat.java:142)
at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1469)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1469)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at cascading.flow.hadoop.planner.HadoopFlowStepJob.internalNonBlockingStart(HadoopFlowStepJob.java:107)
at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:196)
at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:149)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:124)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:43)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

[galarragas]

Hi, this is a known issue.
We are releasing a version for CDH5.1 that supports a more recent version of HBase 0.98.6. Just change your dependency to parallelai:parallelai.spyglass:2.10_0.10_CDH5_4.4

[du291]

@galarragas does that version have autoflush turned off or configurable? (see #12 )
if not which patch should we apply to fix #17 on top of our custom fork that fixes #12 ?

[galarragas]

Hi du291,

The fix you mention is not available in any of the branches. If you want to do a pull request for it on the CDH5 branch I will take care to merge it on both branches.

Cheers

• Stefano

[du291]

@galarragas i don't really have a good patch. We just replaced autoflush(true) with autoflush(false). As I understand point of #12 is to make it configurable. So we'll just pull the CDH5 branch and hotfix it locally.

[galarragas]

Ok, that's a good idea. Let me know if the CDH version is working fine for you. I'm closing this issue

[du291]

@galarragas
I tried the CDH5 branch, but now I get a different error (thrown from the map task attempt):

Error: java.lang.NullPointerException at parallelai.spyglass.hbase.HBaseRecordReaderBase.setHTable(HBaseRecordReaderBase.java:64) at parallelai.spyglass.hbase.HBaseInputFormatGranular.getRecordReader(HBaseInputFormatGranular.java:373) at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:253) at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:248) at cascading.util.Util.retry(Util.java:762) at cascading.tap.hadoop.io.MultiInputFormat.getRecordReader(MultiInputFormat.java:247) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:172) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:414) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1469) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

[galarragas]

Can you share the code causing the exception? I will have a look at it...

Thanks

[galarragas]

I opened issue #18 to track it

[chidu8]

Hi, changing dependency to parallelai:parallelai.spyglass:2.10_0.10_CDH5_4.4 will cause other (major) issues. For example i get an error that the following object is not found: "parallelai.spyglass.hbase.HBaseSource"