Fix flaky MessageIdTest and introduce some testing improvements

[lhotari]

Motivation

MessageIdTest class contains 2 of the flaky test cases in the Pulsar code base:

• org.apache.pulsar.client.impl.MessageIdTest.testChecksumVersionComptability and
• org.apache.pulsar.client.impl.MessageIdTest.testChecksumReconnection

These test cases don't have much to do with MessageId, but are tests for validating message checksum handling in cases where there are pre 1.15 version brokers and post 1.15 version brokers in a mixed broker environment. The tests might not be very relevant any more. However it was taken as a learning experiment to fix these tests and refactor them so that the flakiness of the test code would be eliminated. Similar patterns might be needed in other tests to eliminate flakiness.

Modifications

The changes aren't only to fix MessageIdTest. Most changes could help reduce flakiness of other tests as well.

Improve shutdown of the broker and related services to reduce test flakiness

• await for termination of executors
• close the listen channel synchronously
• use shutdown instead of shutdownNow in AbstractMetadataStore.close
  so that in-flight tasks get processed

Handle special case where the executor rejects the task and the callback was never called

Improve logging in MockedPulsarServiceBaseTest related to stopping and starting

Refactor PulsarClient initialization and lifecycle management in tests

Add getter and setter to access remoteEndpointProtocolVersion field

• it makes it easier to override for tests

Add hooks for overriding the producer implementation in PulsarClientImpl

• useful for tests. Instead of relying on Mockito, there's a pure Java
  way to inject behavior to producer implementations for testing purposes

Introduce PulsarTestClient that contains ways to prevent race conditions and test flakiness

• provides features for simulating failure conditions, for example
  the case of the broker connection disconnecting

Add solution for using Enums classes as source for TestNG DataProvider

Fix flaky MessageIdTest and move checksum related tests to new class

[eolivelli]

great work !

[eolivelli]

@codelipenghui @zymap @sijie @rdhabalia this patch is going to fix the most Flaky test in the suite.

please take a look

thank you @lhotari for contributing this work

[merlimat]

Very nice work, just a couple of comments on executors

[zymap]

nice work!

[merlimat]

@lhotari It seems there's a genuine test failure:

Error:  Failures: 
Error:  org.apache.pulsar.client.impl.PartitionedProducerImplTest.testCustomMessageRouterInstance(org.apache.pulsar.client.impl.PartitionedProducerImplTest)
[INFO]   Run 1: PASS
Error:    Run 2: PartitionedProducerImplTest.testCustomMessageRouterInstance:99->getMessageRouter:105 ? NullPointer
[INFO] 
Error:  org.apache.pulsar.client.impl.PartitionedProducerImplTest.testRoundRobinPartitionMessageRouterImplInstance(org.apache.pulsar.client.impl.PartitionedProducerImplTest)
[INFO]   Run 1: PASS
Error:    Run 2: PartitionedProducerImplTest.testRoundRobinPartitionMessageRouterImplInstance:89->getMessageRouter:105 ? NullPointer
[INFO] 
Error:  org.apache.pulsar.client.impl.PartitionedProducerImplTest.testSinglePartitionMessageRouterImplInstance(org.apache.pulsar.client.impl.PartitionedProducerImplTest)
[INFO]   Run 1: PASS
Error:    Run 2: PartitionedProducerImplTest.testSinglePartitionMessageRouterImplInstance:80->getMessageRouter:105 ? NullPointer
[INFO] 

[lhotari]

It seems there's a genuine test failure:

@merlimat Thanks for the heads up. I'll address it tomorrow.

[eolivelli]

LGTM

[lhotari]

/pulsarbot run-failure-checks