[improve][broker] PIP-444: Rate limit for deleting ledger to alleviate the zk pressure.

[thetumbled]

PIP: #24771

Motivation

When the retention of a large topic is reduced, a significant number of ledgers need to be deleted. Since this deletion operation is not rate-limited, it results in ZooKeeper (ZK) latency of several minutes, as shown below:

Modifications

Add rate limit feature for deleting ledgers.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: thetumbled#75

[Denovo1998]

LGTM.

fixed.

[codelipenghui]

We do have batch operation to the metadata service. If the deletion option happened concurrently, it should be already batched in one request?

[thetumbled]

We do have batch operation to the metadata service. If the deletion option happened concurrently, it should be already batched in one request?

I am afraid that there is no batch feature for bookKeeper.asyncDeleteLedger. Do you know anything related to this? @lhotari @BewareMyPower

[BewareMyPower]

I think @codelipenghui mentioned this feature: #13043, while asyncDeleteLedger involves a metadata store operation.

But from my perspective, the batching feature seems to be enabled by default, so I don't think it can resolve the issue mentioned in PIP-444

[thetumbled]

I think @codelipenghui mentioned this feature: #13043, while asyncDeleteLedger involves a metadata store operation.

But from my perspective, the batching feature seems to be enabled by default, so I don't think it can resolve the issue mentioned in PIP-444

Only when the zk request is issued by broker can this feature takes effect and batch them to alleviate the zk pressure.

However, in this case, the zk requests are sent by the bookie client. Therefore, I'm afraid feature #13043 cannot help resolve this issue.

[BewareMyPower]

Please update the original document #24771 as well, then you can start the vote

[thetumbled]

Please update the original document #24771 as well, then you can start the vote

Updated, and here is the link to the vote thread: https://lists.apache.org/thread/2dop6kyvrpt6ztz9pvylyl8o2syvh9kw.
Thanks.