METRON-2038 Enrichment Loader Fails When Run as MR Job

[nickwallen]

The enrichment loader fails when run as an MR job on YARN. It runs successfully when run in local mode. See METRON-2038 for more information.

Changes

#1345 unexpectedly changed how Maven packages the metron-data-management JAR even though #1345 was completely unrelated to metron-data-management.

Why did #1345 impact metron-data-management?

This is due to the problematic dependency on metron-enrichment from other projects like metron-data-management. This can cause dependency changes in unrelated projects like metron-profiler-client to impact completely unrelated projects like metron-data-management. This problematic dependency chain is in the process of being cleaned up as part of other ongoing work.

Why did the enrichment loader not work in MR, but work in local mode?

The problem occurs because Maven packaged org.apache.hadoop.hbase.HBaseConfiguration in the metron-data-management jar, even though HBase is marked as a provided dependency. This really should not be packaged in the JAR, but I have been unable to determine why Maven was doing this.

When the enrichment loader is executed by the flatfile_loader.sh script, a large number of HBase dependencies are added at runtime. The underlying MR code requires a different, conflicting version of this class. Unfortunately, it would find the conflicting org.apache.hadoop.hbase.HBaseConfiguration class definition in the metron-data-management jar rather than in its runtime dependencies.

How was this fixed?

Rather than load the HBase dependency at runtime, now we package the HBase dependency in the JAR and do not add any of these dependencies at runtime. This approach avoids the conflict and helps ensure that the integration tests run more similarly to how the tool is used in production.

Testing

Follow the steps to replicate from METRON-2038 in Full Dev. Ensure that the enrichments are loaded to the enrichments table in HBase.

Pull Request Checklist

• Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
• Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
• Has your PR been rebased against the latest commit within the target branch (typically master)?
• Have you included steps to reproduce the behavior or problem that is being changed or addressed?
• Have you included steps or a guide to how the change may be verified and tested manually?
• Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
• Have you written or updated unit tests and or integration tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

[mmiklavc]

We should probably reference this test script for regression testing
#432 (comment)

[mmiklavc]

Ah, this seems like it might be related - https://issues.apache.org/jira/browse/MSHADE-267. I also saw that HBaseConfiguration is included in the uber jar. It looks like it's getting this from hbase-common, which is marked as provided, but may be sneaking in due to transitive dependencies of test jars that include it in compile scope.

[mmiklavc]

@nickwallen - Thanks for tackling this. I happened to also see the rogue HBaseConfiguration along with a number of other deps that come from hbase-common getting pulled into the metron-data-management uber jar. Bootstrapping off of your findings, I thought it worthwhile to dig into this a bit more. It seemed like there might be a bug in the Maven shade plugin which would potentially cause us more trouble down the line and I wanted to see what we were in for. For starters, here's the class I'm using as a tracer bullet.

jar tvf /Users/mmiklavcic/.m2/repository/org/apache/metron/metron-data-management/0.7.1/metron-data-management-0.7.1.jar|grep HBaseConfiguration
  7195 Mon Mar 18 12:02:08 MDT 2019 org/apache/hadoop/hbase/HBaseConfiguration.class

I went through a similar series of inclusions and exclusions in the mdm pom.xml only to find no change in the rogue deps appearing. Also, regardless of their scope tag, I continued to see that classes in the uber jar. I got to the point where I had excluded all transitive dependencies for hbase-common and still was seeing the classes. Either there was a bug in the shade plugin or we were missing something.

The next step I took was to look over the dependency:tree from the CLI - all deps checked out accordingly. Default scope included by the Shade plugin? - yep, scope "provided" is not included by default. I also examined the build output from the Shade plugin - very clearly, there is no hbase-common dependency included in the shading. This is starting to feel like The Adventures of Buckaroo Banzai Across the 8th Dimension or The Adventures of Baron Munchausen. Either way, it's a trip.

Ok, let's go more brute force on this and explicitly copy dependencies. I looked at the compile scope dependencies first, but that didn't look quite right. I landed on runtime scope most closely representing what the Shade plugin bundles. I then iterated over all the jars looking for the config class.

mvn dependency:copy-dependencies -DoutputDirectory=/tmp/mikeout -DincludeScope=runtime
for jar in /tmp/mikeout/*.jar; do echo $jar; jar tvf $jar|grep HBaseConfiguration; done
...
/tmp/mikeout/metron-profiler-client-0.7.1.jar
  7161 Fri Mar 15 15:45:58 MDT 2019 org/apache/hadoop/hbase/HBaseConfiguration.class
...

Ok, so there it is. But why??? I looked over our dependency graph and see that metron-data-management depends on metron-enrichment depends on metron-profiler-client. Ok, there's the link, but what about hbase-common? metron-profiler-client has a dependency on hbase-common, scope=compile. Getting closer. The client's pom also has a reference to the maven shade plugin and an assembly.xml. I explored dependencies on metron-profiler-client from a shell script and deployment perspective and did not find any circumstances where we would need to deploy this module stand-alone. I bumped the hbase-common dependency to provided (debatable) and removed all references to shading/relocating and a tarball assembly. I rebuilt the project and no longer see the HBaseConfiguration, or any other hbase-common classes for that matter, in the metron-data-management jar.

I'm currently undertaking testing from #432 (comment) as indicated above, but I think this should solve the root problem without requiring us to modify our license file or eliminate the core Hadoop libs from our shell script classpath. I've created a PR against your PR here - nickwallen#15

[mmiklavc]

As I worked through the full gamut of testing, all of the flatfile loader tests work, but it re-introduces the exception found in #1345.

I would like to explore this a bit further in a separate PR to see if I can get this working, but for now I'm +1 on the current work.

[nickwallen]

Wow - nice investigation. I think we definitely need to clean-up what you've identified. I think a follow-on for that effort will work.