Downgrade the version of Apache Curator from 5.5.0 to 5.3.0 to avoid a bug in the new version

[asdf2014]

Fixes #16411

Release note

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[cryptoe]

I think this is important enough to hold the release for druid 30.

[gianm]

Are there any fixes in Curator 5.4 or 5.5 for other bugs that would be important for us? I ask since if yes- we could stay with the newer Curator and work around this bug, such as by closing and recreating our LeaderLatch when ZK session changes.

[gianm]

There could be a comment here about why we're in this version, so the rationale won't get forgotten. (Link to the Curator JIRA is best)

[asdf2014]

@gianm Done. FYI, #16444

[gianm]

I haven't reviewed these lists yet but here they are-

• 5.4 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12352051
• 5.5 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12352495
• 5.6 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12353185

[gianm]

Just reviewed the lists. By downgrading from 5.5 to 5.3 we do lose various fixes. These ones sound like they could be important:

• [CURATOR-504] - Race conditions in LeaderLatch after reconnecting to ensemble
• [CURATOR-638] - Curator disconnect from zookeeper when IPs change [seems especially relevant to k8s environments]
• [CURATOR-644] - CLONE - Race conditions in LeaderLatch after reconnecting to ensemble [a live-lock issue; we believe the fix for this bug introduced the split-brain problem; so rolling back would reintroduce the live-lock issue]
• [CURATOR-649] - Background exception was not retry-able or retry gave up [robustness]

With regard to downgrading Curator to 5.3 in Druid 30, I think we should be especially careful of these issues, and in particular CURATOR-638 and possible impact on k8s environments.

Alternate approaches do include:

• Stay with Curator 5.5 and work around this bug, such as by closing and recreating our LeaderLatch when ZK session changes.
• Stay with Curator 5.5 and don't work around this bug; wait for release of Curator 5.7 which will hopefully include a fix.