[fix][flaky-test]ManagedCursorMetricsTest.testManagedCursorMetrics

[poorbarcode]

Fixes #9919

Master Issue: #9919

Motivation

This problem was marked as resolved, but it still exists.

recent failure: https://github.com/apache/pulsar/runs/7564370916?check_suite_focus=true#step:10:950

and this one: https://github.com/apache/pulsar/pull/16841/checks?check_run_id=7566225112

Why is this test flaky？

• The ledgerHandle of the cursor, is not accurately closed, even if it is closed, there is no guarantee that ledgerHandle created will occur after cursor.persistentToLedger.

pulsar/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/ManagedCursorMetricsTest.java

Lines 98 to 100 in 2d6b534

	for(PulsarMockLedgerHandle ledgerHandle : mockBookKeeper.getLedgerMap().values()) {
	ledgerHandle.close();
	}

• This line of below could not guarantee that cursor.persistentToLedger has been completed, only that the client has received a response.

pulsar/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/ManagedCursorMetricsTest.java

Line 108 in 2d6b534

Awaitility.await().until(() -> pulsarClient.getConnection(topicName).get().getPendingRequests().size() == 0);

Modifications

These following changes in this PR (High light):

1. Fixed flaky test ManagedCursorMetricsTest.testManagedCursorMetrics.
2. Add more test cases. Only brk_ml_cursor_persistZookeeperErrors is left incomplete now, I write the reason in the code comment.
3. Improves performance, use @BeforeClass instead of @BeforeMethod(alwaysRun=true)

Documentation

• doc-required
  (Your PR needs to update docs and you will update later)

• doc-not-needed
  (Please explain why)

• doc
  (Your PR contains doc changes)

• doc-complete
  (Docs have been already added)

[poorbarcode]

/pulsarbot rerun-failure-checks

[poorbarcode]

@lhotari Could you take a look ?

[Jason918]

@poorbarcode Can you help open a new PR to merge this to branch-2.10?

[poorbarcode]

Hi @Jason918

Can you help open a new PR to merge this to branch-2.10?

yes, it is #17504. Note that the branch-master logic changes cause the test code for branch-2.10 to be a little different