METRON-495: Upgrade Storm to 1.0.x

[justinleet]

***** INCOMPLETE DO NOT MERGE *****

Putting the PR out, so people can take a look and potentially fix/improve things.

Initial start at upgrading the Storm version to 1.0.1, and I'm going to hop off of this for right now. Provided in case anyone wants to take a stab at fixing / cleaning up the various issues with it, and so feedback can be provided. If you do make improvements, feel free to grab the branch and submit a new PR.

What changes occurred

• The move over to org.apache.storm from backtype.storm
• API changes from byte[] to ByteArray are handled
• Kafka also updated to 0.10 out of necessity, storm-kafka is still used (storm-kafka-client seemed like significantly more work given the amount of changes to the underpinnings). I would not expect this to cause issues, given backwards compatibility, but since tests still fail this hasn't been validated.
• Some of the various classpath issues have been taken care of (things move between jars, and a couple test items now need a test jar which has been pulled in and scoped appropriately on various modules)
• A couple simple Scala wrappers about some of the Storm Kafka handling in regards to Zookeeper has been added (including to the pom) to avoid having to call direct Scala only testing tools that use pretty Scala specific stuff. The wrappers just take in the args we were using and use defaults for the rest.
• Changes to the full dev platform to account for using HDP 2.5 to match the new version. This is entirely untested given test failures.

What works

• Most unit tests. Interestingly, metron-platform/metron-management fails on CLI, but succeeds in IntelliJ. I'm guessing classpath issue as a result of attempting to fix integration tests broke this on the CLI as it used to work, but unsure.
• Most integration tests. This involved moving.

What doesn't work

• Both some unit and some integration tests
• Dev platforms are untested.

What's left

• Fixing tests
• Testing full dev
• Building quick dev image and testing it
• End to End testing. Given the scope of the changes, this needs to be pretty thorough.
• Ambari MPack testing.

[justinleet]

As an aside, the reason for attempting to bump the version of Kafka is that there seems to be a bit of a catch where moving Storm to 1.0.1 causes problems with Kafka, but leaving it obviously means the packaging no longer lines up. If someone knows a way around that (or even better, a more correct way to handle that) without upgrading Kafka, I'd probably prefer to use that, and leave Kafka for another day.

[ottobackwards]

Justin -
Just to clarify, you see mvn test work in intellij but fail from the cli?

[ottobackwards]

I just did two mvn integration-test runs back to back.

First run : parsers failed
Second run: parsers passed, metron-pcap-backend failed

[ottobackwards]

I spent some time on this yesterday. I do not have a lot of Kafka/ZK/Storm experience - but I was hoping for something to stand out....
Unfortunately that was not the case. Can you share how you setup intellij for scala? I'm seeing intellij not finding the wrapper classes.
Also - if you have any more information on the failing tests or guesses as to their cause that would help.

Do we know if we should have a newer version of curator to go with the newer kafka/storm?

[justinleet]

Closing this PR, as it's been superseded by Mike and Casey's follow on work at #318