SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role

[gerlowskija]

Description

Prior to this commit, Solr only supported running embedded-ZK in "standalone" mode (which cannot take part in any larger ZK ensemble or quorum). But there are usecases that would benefit from being able to do this, both on the development and testing side, and event for adventurous users who might want the benefits of a small multi-node SolrCloud cluster without the headache of also deploying ZK.

See SIP-14, SOLR-15636 for context

Solution

This commit augments our embedded-ZK code to support running embedded-ZK
in "quorum" or ensemble mode. Multiple Solr nodes can now all have
their embedded-ZK's join a multi-node quorum upon startup. Other than
Solr and ZK sharing a process, the embedded- ZK ensemble behaves
identically to one formed of independent processes: nodes can join or
leave the cluster, etc.

Embedded-ensemble-ZK is enabled any time the zkQuorumRun system
property is present, along with an explicitly specified ZK host string.
On startup, Solr will identify which host in the zk-conn-string it
should be (based on admittedly hacky heuristics), and then spins up a
'ZooKeeperServerEmbedded' instance in-process to join the ensemble. e.g.

export LH="localhost"

# Run each below in a separate terminal, one for each host in the zk-conn string
bin/solr start -p 8983 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun
bin/solr start -p 8984 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun
bin/solr start -p 8985 -z $LH:9983,$LH:9984,$LH:9985 -DzkQuorumRun

Some notes:
- this doesn't (yet) work with ZK's dynamic-ensemble feature, so all
ZK nodes must be specified in a static ZK conn string provided at
startup
- this appears to run best when the security-manager is disabled.
- the interface in particular for how we expose this is pretty rough, and there's a lot of room for improvement.

Tests

• TestEmbeddedZkQuorum

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide

[gerlowskija]

FYI: https://cwiki.apache.org/confluence/display/SOLR/SIP-14+Embedded+Zookeeper for context on others' interest in moving this direction.

[janhoy]

Great work. I think we should rip out the existing embedded ZK, which is a hack.

Do you want to proceed in a PR or perhaps in a central feature branch?

[gerlowskija]

Spent a few minutes pulling in the latest 'main'. Hoping to give it a bit more cleanup in the next few days if I can, but I've been hoping that for more than a year at this point (😬 ), so if anyone else is interested please feel free to move this forward!

[epugh]

@gerlowskija I took a stab at getting in the use of the solr roles to determine if we hsould run embedded zk quorum mode, and it worked! Here is my start script, where each runs with -f so you need three terminals. Omit the -f and you don't need the seperate terminals.

mkdir -p ./three_nodes/node1/solr
mkdir -p ./three_nodes/node2/solr
mkdir -p ./three_nodes/node3/solr

SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8983 --solr-home "./three_nodes/node1/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8984 --solr-home "./three_nodes/node2/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on
SOLR_SECURITY_MANAGER_ENABLED=false bin/solr start -f -p 8985 --solr-home "./three_nodes/node3/solr" -z localhost:9983,localhost:9984,localhost:9985 -Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on

[janhoy]

• Merged in main branch
• Properly clean up the ZK server resource in close()

[epugh]

FYI, the .bats test worked a few times for me and then quit working ;-(

[epugh]

I asked Claude to summarize the last seven commits and he said:

Recent work on the spike-zk-quorum branch has focused on refining Solr's embedded ZooKeeper quorum mode implementation through targeted code improvements. The codebase has been streamlined by removing unnecessary variables and conditional logic in the SolrZkServer class, particularly eliminating redundant checks around the zkServerEnabled flag and renaming methods like startZKSE to more descriptive alternatives. Initialization has been optimized by removing the /solr/initialized ZooKeeper node, which was only needed for multi-threading protection that's unnecessary in the current implementation. The ZooKeeper initialization logic in ZkContainer has been restructured with clearer nesting and better separation between standalone and quorum modes, making the codebase more maintainable. These changes collectively enhance the embedded ZooKeeper quorum functionality while simplifying the code structure, resulting in a more robust and easier-to-understand implementation that properly manages ZooKeeper server resources throughout Solr's lifecycle.

I think my summary is: I discovered that we have TWO ways of starting ZK, not including our independent ZkTestServer that runs a ZkTestServer.ZKServerMain during tests.

We now have:

• For testing only, we have org.apache.solr.cloud.ZkTestServer.ZKServerMain that wraps org.apache.zookeeper.server.ZooKeeperServer.
• org.apache.solr.cloud.SolrZkServer that wraps either a org.apache.zookeeper.server.ZooKeeperServer OR if there is more then one, then does a org.apache.zookeeper.server.quorum.QuorumPeerMain
• org.apache.solr.core.ZkContainer that starts up org.apache.zookeeper.server.embedded.ZooKeeperServerEmbedded

I am now wondering could we convert SolrZkServer to using a ZooKeeperServerEmbedded instead of a ZooKeeperServer?

Secondly, now seeing the use of QuorumPeerMain, which currently kicks in if there are multiple ZooKeeper servers defined (presumably only in an external cluster set up), I wonder if we need to rename roles: zookeeper_quorum to zookeeper. If you have zookeeper then you fire up ZooKeeperServerEmbedded. And if not, then you default to QuorumPeerMain.

Thirdly, why is ZkContainer in o.a.s.core and not o.a.s.cloud?

I did take a stab at the SolrZkServer using the ZooKeeperServerEmbedded, updating the start/stop methods, but failed ;-(. Going to need to put this down for a bit.

[janhoy]

I did take a stab at the SolrZkServer using the ZooKeeperServerEmbedded, updating the start/stop methods, but failed ;-(. Going to need to put this down for a bit.

Thanks for chipping away on this. At some point we'll have something that can be released in 10.1 or 10.2 as an experimental feature for folks to try out, without dynamic reconfig. Then we can tackle the dynamic stuff...

[epugh]

Thanks for taking a look at this @janhoy I think I've stalled a bit on this... It feels "so close' yet not sure how to get it over the hump.

[janhoy]

Thanks for taking a look at this @janhoy I think I've stalled a bit on this... It feels "so close' yet not sure how to get it over the hump.

Yep, I got it up to speed with main, hardened the qourum test, and try to add test for stopping a quorum node and resume operation.

I think we need to focus on the startup modes / params to have a clear story for the three modes as you mentioned

• --user-managed
• -c (implicit) with one embedded ZK
• -z external:2181 for existing external zk cluster
• Managed embedded quorum

For the managed embedded quorum, I believe we have proved that setting the node role explicitly and passing -z to all nodes pointing to those quorum nodes will do the trick. That is probably good enough for a first release?

Phase 1:

Ship a experimental version with a working zk node-role, that starts a quorum that is somewhat stable.

Phase 2:

Make it easier to setup for a small test cluster with some convenience flags (see above). integrate into solr operator.
Enable ZK ACL stuff by default.

Phase 3:

Good support for SSL between solr and embedded zk quorum. Either self-provided certs, or self-signed certs during startup. Note, we can have non-ssl Solr API while still having SSL between solr and zookeeper.

Phase 4:

Support dynamic reconfiguration. I don't know how important that really is. In k8s service-names are stable and you don't need to reconfigure zk really, except when changing the size of your quorum. But in self managed infrastructure, you may want to retire some servers and move zk over to another set of nodes by reconfiguring in two phases.

[epugh]

I wonder if the -c mode and the Managed embedded quorum are just the same? Maybe it's "the first three nodes that get fired up are all zk nodes".

[janhoy]

I wonder if the -c mode and the Managed embedded quorum are just the same? Maybe it's "the first three nodes that get fired up are all zk nodes".

I don't like that kind of magic. I hope we can introduce this as an additional mode, then once it proves itself and is stable, we can make this mode the default when just starting a single node. But then we're at phase 4, where we have some degree of dynamic configuration and support for growing a quorum. I don't think that is important in the first few releases.

I come to value stability and some degree of static non-mutable config. So it is not a crisis if you need to specify the ZK_HOST with all your quorum nodes when starting a solr cluster. It gives you clarify and stability. You don't get the cool dynamic stuff but in large clusters you setup quorum nodes once, either 3 or 5 and then don't touch them for years. All your 50 solr data nodes will have that exact same ZK_HOST string. And if you change it, well, you'll have a fun rolling restart of the entire cluster.

[epugh]

I wonder if the -c mode and the Managed embedded quorum are just the same? Maybe it's "the first three nodes that get fired up are all zk nodes".

I don't like that kind of magic. I hope we can introduce this as an additional mode, then once it proves itself and is stable, we can make this mode the default when just starting a single node. But then we're at phase 4, where we have some degree of dynamic configuration and support for growing a quorum. I don't think that is important in the first few releases.

I come to value stability and some degree of static non-mutable config. So it is not a crisis if you need to specify the ZK_HOST with all your quorum nodes when starting a solr cluster. It gives you clarify and stability. You don't get the cool dynamic stuff but in large clusters you setup quorum nodes once, either 3 or 5 and then don't touch them for years. All your 50 solr data nodes will have that exact same ZK_HOST string. And if you change it, well, you'll have a fun rolling restart of the entire cluster.

that makes sense... I suspect you have a lot more experience in running clusters at scale than I do!

[janhoy]

@gerlowskija I removed NOCOMMIT in this comment to unblock other precommit tests. Making this review comment to make sure we don't forget about it :)

[janhoy]

I created SOLR-18094 as jira for this issue (sub-task of the SIP JIRA).

Let's focus on getting this PR in shape for releasing the building blocks of the Zk code and the node-role. Target 10.1 as an experimental feature, can document it as such and solicit feedback from users.

I made Precommit run, will spend some time some day to try to do a more thorough review. Let's make review comments for all known rough edges and things lacking.