Patform now buildable from scratch

[djspiewak]

Eliminated some dependencies and corrected things such that it is actually possible to build on a clean machine. Note: we're still dependent on a version of Kafka which was never published, mostly because the Kafka project is extremely fail and has never published anything. To get around this, I have created a tarball of the files needed. This tarball, along with the full build instructions, is linked in the readme.

[jdegoes]

@djspiewak This looks good to me. Did you verify that the older version of HOWL doesn't break things?

[guillaug]

It seems that you still have dependencies on your private s3 bucket containing zookeeper and kafka zip files : when running start-shard.sh

zookeeper link can be easily edited by redirecting to an apache mirror but the full zip of the unreleased kafka-0.7.5 is still an issue... Could you point to a commit in the kafka repo ? So we can build the zip ourself ?

[djspiewak]

@djspiewak This looks good to me. Did you verify that the older version of HOWL doesn't break things?

I haven't yet. It was getting late when I got to it. As far as I can tell, the custom version of HOWL only provides a setScheduler method on an existing configuration object. This allows HOWL to share a thread pool with NIHDB. As far as I know, this is just a performance optimization. I did of course verify that it compiles and the test suite is able to run. Getting that far actually verifies more than you would think, since it requires NIHDB to be in working order along with bifrost and shard.

It seems that you still have dependencies on your private s3 bucket containing zookeeper and kafka zip files : when running start-shard.sh

Ugggghhhh… Could you open another issue? I don't even know if we still have that S3 bucket active, and so we may not have those files on hand.

As for the Kafka commit, I wish it were as simple as pointing to a commit and giving build instructions. That's what I was trying to do at first. Unfortunately, the commit is on an old version of Kafka which actually doesn't build! I tried for about an hour, but it steadfastly refused to compile cleanly. I'm pretty sure it has dependencies on internal segments of LinkedIn's build infrastructure. This would explain why the Precog custom version was labeled 0.7.5, rather than 0.7.2 (which is the latest in the 0.7 stream). I think we probably ported forward the build and fixed the compilation errors so as to be able to publish a JAR. Those commits may exist somewhere, but I couldn't find them.

To the best of my knowledge, the only way to get the working Kafka JAR is to use my tarball, and the only way I was able to get that tarball was to literally slurp the files out of my Ivy cache. Given this, excising Kafka and Zookeeper should probably be the highest development priority.