[Issue 12812] Introduce ZKBatchedMetadataStore.

[Jason918]

Fixes #12812

Motivation

See #12812

Modifications

Add class ZKBatchedMetadataStore extends ZKMetadataStore and overrides storeGet and getChildrenFromStore.
Put these two kinds of read operations into org.apache.pulsar.metadata.impl.batch.BatchWorker#opQueue, which contains a background thread that will send batch ops once it reaches target op number (set as 100) or waited long enough (5ms).

Verifying this change

• Make sure that the change passes the CI checks.

This change is already covered by existing tests, all test cases in pulsar-metadata.

This change added tests and can be verified as follows:

• ZKBatchedMetadataStoreTest

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): (no)
• The public API: (no)
• The schema: (no)
• The default values of configurations: (no)
• The wire protocol: (no)
• The rest endpoints: (no)
• The admin cli options: (no)
• Anything that affects deployment: (no)

Documentation

Check the box below and label this PR (if you have committer privilege).

Need to update docs?

• no-need-doc

Pref optimization.

[Jason918]

/pulsarbot run-failure-checks

[Jason918]

/pulsarbot run-failure-checks

[Jason918]

/pulsarbot run-failure-checks

[merlimat]

This is implementing the batching for getData/getChildren calls but not for the create and setData which will be very interesting, even though we need to solve the problem of creating the parent nodes.

I'll submit a different PR with the logic for the write/create

[merlimat]

Instead of extending the ZKMetadataStore, I believe we should have a common abstract class that implements the batching logic, which could be shared across multiple implementations.

[Jason918]

I wrote a AbstractBatchedMetadataStore at first, but got some issue.

• We have to use two different batching queues for Zookeeper Multi operations, one for read and one for write. But I believe that we don't need to do that in other metadata store implementations, e.g. Memory, RocksDB or maybe etcd version of implementation.
• There are not many things left in this AbstractBatchedMetadataStore, mostly is in wrapped in BatchWorker, maybe only fallbackToSingleOps.

[merlimat]

These parameters should be left configurable, in particular the enable/disable, as we might want to disable in case we see bugs.

[Jason918]

OK, I will fix this.
@codelipenghui @hangc0276 FYI.

[Jason918]

This is implementing the batching for getData/getChildren calls but not for the create and setData which will be very interesting, even though we need to solve the problem of creating the parent nodes.

I'll submit a different PR with the logic for the write/create

This is WIP. Adding create and setData is pretty easy based on this PR.

I plan to get a comparison result of containing batch read operations first, as we can see in previous zk perf result #12812, batch reading have the most performance improvements.

[merlimat]

@Jason918 I've added a different PR based on the approach I was mentioning: #13043. PTAL

[Jason918]

@Jason918 I've added a different PR based on the approach I was mentioning: #13043. PTAL

OK. This seems to be a duplicated work. Let's continue this work on your PR.