Deprecate support for Apache ZooKeeper 3.4.x

[xvrl]

ZooKeeper 3.4.x has been EOL for some time, and its client libraries are no longer supported by the community.
In order to support newer JDK versions (i.e. JDK 14, 15 and above) Druid will need to upgrade to 3.5.x client libraries (see https://issues.apache.org/jira/browse/ZOOKEEPER-3779)

Unfortunately Apache Curator does not support ZooKeeper 3.4.x with newer clients libraries (see https://curator.apache.org/zk-compatibility-34.html) and as such we will need to drop 3.4.x support in order to upgrade.

I propose we deprecate support for ZooKeeper 3.4.x in the next release, and remove support in a subsequent one, once we validate that 3.5.x client libraries perform as expected.

[gianm]

What would the migration path look like? Would people need to update Druid and their ZK quorum in a specific order or would various orders works? Any downtime needed?

The Curator link says:

If you wish to use Curator with ZooKeeper 3.4.x you should pin to version 4.2.x of Curator.

But I'm not sure if this refers to a 3.4.x client or a 3.4.x server (or both).

[himanshug]

If 3.5.x ZK server is backward compatible with 3.4.x client then we can update ZK server version in Druid integration test builds to ensure that it works as expected.

[clintropolis]

What would the migration path look like? Would people need to update Druid and their ZK quorum in a specific order or would various orders works? Any downtime needed?

I am assuming since newer client libs do not support 3.4.x that we would announce deprecation while continuing to use the existing client libs, to give time for operators to migrate clusters to 3.5.x. So, (if I understand correctly) the path initially would only involve upgrading ZK, which I would think could be done out of band with any Druid updates.

Assuming this is correct, I think this should be relatively low risk to encourage cluster operators to upgrade their ZK cluster to 3.5.x (other than the risk of doing the upgrade of course); my anecdotal evidence for this being that I have been using it locally for all of my testing since shortly after 3.5.x became a 'stable' branch and haven't ran into any issues using the current set of client libraries.

But I'm not sure if this refers to a 3.4.x client or a 3.4.x server (or both).

Based on this comment in our pom.xml about zk version, I assume client libraries?

[clintropolis]

If 3.5.x ZK server is backward compatible with 3.4.x client then we can update ZK server version in Druid integration test builds to ensure that it works as expected.

In ZK 3.6.0 release notes (which I can't directly link to a specific one unfortunately), it states

ZooKeeper clients from 3.4 and 3.5 branch are fully compatible with 3.6 servers.

I didn't see anything explicitly stating this for the 3.5.5 release notes (when 3.5.x became "stable"), but I assume it holds true.

Updating the integration tests to use ZK 3.5.x (or 3.6.2? the downloads page seems to indicate this is the latest stable version) seems like a good first step to me.

[gianm]

Based on what y'all are saying, it sounds like the migration plan would be:

1. We announce that we're going to be removing support for ZK 3.5 servers in the future, but don't actually do anything yet, except use ZK 3.5 in integration tests to make sure everything works right
2. Users should update their ZK servers to 3.5. This should work with the ZK 3.4 client that we use
3. At some point in the future we switch to the ZK 3.5 client. People must upgrade the ZK server to 3.5 before upgrading to this new Druid version

[xvrl]

@gianm yes, the migration path is to first upgrade ZK 3.5 before upgrade Druid to whichever version will no longer support ZK 3.4.x. We have been running Druid 0.19 successfully with both 3.4.x and 3.5.x ZK clusters, so any recent Druid version should run fine on either ZK version.

[himanshug]

We have been running Druid 0.19 successfully with both 3.4.x and 3.5.x ZK clusters

since there is a clear and tested migration path, I agree with everything in #10780 (comment)

[gianm]

In that case, I suggest add the tests for ZK 3.5 now, and we announce in the very next release that people should upgrade to ZK 3.5 when they can, because a future release will ship with the ZK 3.5 client, and will no longer support ZK 3.4 servers.

[jihoonson]

It makes sense to me to deprecate 3.4 in 0.21.0.

[gianm]

Should we go to 3.5.x or 3.6.x?

Given that both are supported, I'm leaning towards 3.5.x, because it's more likely to be battle-tested / stable. That's mostly what we want from ZooKeeper anyway. It should also maximize server compatibility, since it seems that older ZK clients typically work well with newer servers.

[clintropolis]

Should we go to 3.5.x or 3.6.x?

https://downloads.apache.org/zookeeper/stable/ shows 3.5.9 as stable version, while 3.6.2 is 'current' https://downloads.apache.org/zookeeper/current/, so I agree with going 3.5.x for now