Backward incompatible change with Tranquility

[niketh]

@gianm @cheddar

#4060 is causing a backward incompatible change which is breaking existing Tranquility, here is the error
`2017-03-22 14:02:06.527 y.s.a.s.DruidBeamSink TranquilitySink_[5 5] [ERROR] TranquilitySink - Problem preparing DruidBeam bolt in worker 5: java.lang.RuntimeException: org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "enabled" (Class org.apache.shaded.curator.x.discovery.ServiceInstance), not marked as ignorable
at [Source: [B@1913fa09; line: 1, column: 234] (through reference chain: org.apache.shaded.curator.x.discovery.ServiceInstance["enabled"])``

[gianm]

That looks like we need to set the "ignore unrecognized property" flag on whatever object mapper Tranquility is using, so probably a Tranquility bug rather than Druid bug.

What's the nature of the incompatibility exactly? Do you get that error when you run latest Tranquility against a Druid 0.10.0-rc1 cluster?

[niketh]

@gianm We are using 0.5.0 version tranquility against a Druid latest (master branch - commit 6febcd9)

[gianm]

@niketh, I tried locally and confirmed this is still a bug with the latest tranquility. It's fixed by updating Tranquility's Curator. After doing that, tranquility works against both Druid 0.10.0 and 0.9.2. Probably other versions too!

I'll send a Tranquility PR in a bit.

[gianm]

Tranquillity patch at: druid-io/tranquility#213. @niketh are you able to try that out? You can do a build by running:

sbt ++2.11.8 'project distribution' universal:packageZipTarball

[gianm]

Marking 0.10.0 release notes, this should at least end up there if nothing else. Something along the lines of needing to update tranquility first if you're using that.

[gianm]

Oops, closed accidentally.

[gianm]

I raised https://issues.apache.org/jira/browse/CURATOR-394 with Curator. The root cause of this looks like a Curator bug. In this situation, it should be doable to upgrade Tranquility (the client) first and then Druid (the service), which would work around that Curator issue. It adds to annoyance for users though.

Some questions:

• Should we revert the upgrade to Curator 2.12.0 (Update Curator to 2.12.0. #4060) and wait for this to be sorted out upstream? Curator 2.12.0 is supposed to fix Another day, another Apache Curator bug #4056, Curator leader election breaks in the overlord when zookeeper has issues #3837 so there is some downside to reverting. But the upside is that it makes upgrading Druid a lot simpler for Tranquility users, since they don't need to upgrade Tranquility first.
• If we stick with Curator 2.12.0, this issue means that Curator service discovery clients need to be updated before services. I think for Tranquility we can deal with this, but is there anything else in Druid where it'd be difficult to do this?

[gianm]

My feeling is it makes sense to revert the Curator 2.12.0 upgrade.

[gianm]

Talking with Jordan on CURATOR-394, it sounds like they may revert their change that broke stuff here, and do another Curator release. So for 0.10.0 I think we should revert back to the older Curator, and then revisit upgrading Curator in 0.10.1. I'll do a patch for that in a bit.