MINOR: System tests for Raft-based metadata quorums

[rondagostino]

Adds a new sanity_checks/bounce_test.py system test for a simple produce/bounce/produce series of events. Augments this and other sanity_checks tests so they run all metadata quorum types: ZooKeeper, remote Raft, and co-located Raft.

Augments several tests/client and tests/core system tests to run for remote Raft-based metadata quorums in addition to ZooKeeper. Co-located Raft controllers are not tested here as this configuration is presumed to work assuming sanity_checks as well as any unit/integration tests pass.

Augments a simple set of tests in tests/connect, tests/streams, and tests/tools to run for remote Raft-based metadata quorums in addition to ZooKeeper.

Adds missing @cluster annotations for dozens of system tests that were missing them -- this was causing these tests to grab the entire cluster of nodes and negatively impact the parallelism of the system test run.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[rondagostino]

There are 6 failing tests across these 3 test classes:

kafka.server.KafkaConfigTest

KafkaConfigTest > testFromPropsInvalid() FAILED
    org.opentest4j.AssertionFailedError: Expected exception for property `broker.heartbeat.interval.ms` with invalid value `-1` was not thrown ==> Expected java.lang.Exception to be thrown, but nothing was thrown.

AutoTopicCreationManagerTest

AutoTopicCreationManagerTest > testCreateTxnTopic() FAILED
    Wanted but not invoked:
    brokerToControllerChannelManager.sendRequest(
        CreateTopicsRequestData(topics=[CreatableTopic(name='__transaction_state', numPartitions=2, replicationFactor=2, assignments=[], configs=[])], timeoutMs=100, validateOnly=false),
        <any kafka.server.ControllerRequestCompletionHandler>
    );
    -> at kafka.server.AutoTopicCreationManagerTest.testCreateTopic(AutoTopicCreationManagerTest.scala:119)
    Actually, there were zero interactions with this mock.

AutoTopicCreationManagerTest > testCreateOffsetTopic() FAILED
    Wanted but not invoked:
    brokerToControllerChannelManager.sendRequest(
        CreateTopicsRequestData(topics=[CreatableTopic(name='__consumer_offsets', numPartitions=2, replicationFactor=2, assignments=[], configs=[])], timeoutMs=100, validateOnly=false),
        <any kafka.server.ControllerRequestCompletionHandler>
    );
    -> at kafka.server.AutoTopicCreationManagerTest.testCreateTopic(AutoTopicCreationManagerTest.scala:119)
    Actually, there were zero interactions with this mock.

AutoTopicCreationManagerTest > testCreateNonInternalTopic() FAILED
    Wanted but not invoked:
    brokerToControllerChannelManager.sendRequest(
        CreateTopicsRequestData(topics=[CreatableTopic(name='topic', numPartitions=1, replicationFactor=1, assignments=[], configs=[])], timeoutMs=100, validateOnly=false),
        <any kafka.server.ControllerRequestCompletionHandler>
    );
    -> at kafka.server.AutoTopicCreationManagerTest.testCreateTopic(AutoTopicCreationManagerTest.scala:119)
    Actually, there were zero interactions with this mock.

MetadataRequestWithForwardingTest

MetadataRequestWithForwardingTest > testAutoCreateOfCollidingTopics() FAILED
    org.opentest4j.AssertionFailedError: expected: <Set(LEADER_NOT_AVAILABLE, INVALID_TOPIC_EXCEPTION)> but was: <Set(UNKNOWN_TOPIC_OR_PARTITION)>

MetadataRequestWithForwardingTest > testAutoTopicCreation() FAILED
    org.opentest4j.AssertionFailedError: expected: <LEADER_NOT_AVAILABLE> but was: <UNKNOWN_TOPIC_OR_PARTITION>

[ijuma]

Let's file a JIRA to discuss whether the metadata module should use Kafka metrics or Yammer metrics.

[rondagostino]

file a JIRA to discuss whether the metadata module should use Kafka metrics or Yammer metrics.

https://issues.apache.org/jira/browse/KAFKA-12348

cc: @cmccabe

[cmccabe]

thanks. I think this will be a post-2.8 thing

[rondagostino]

http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2021-02-20--001.1613812911--rondagostino--kip500_full--f59453faa/report.html

Tests|Passes|Failures|Ignored|Time
940|558|185|197|359 minutes

Some observations. First, adding the missing @cluster annotations really helped. Running almost 750 tests in under 6 hours is a dramatic improvement. Here’s the stats from a run 2 days ago:

Tests|Passes|Failures|Ignored|Time
729|427|105|197|388 minutes

We are running 743/532 = 40% more tests, but we are able to finish in 10% less time. So that’s a big win.

We have 130 more tests passing and 80 more tests failing. It is VERY difficult to find the signal amid so much noise. A quick browse through our results shows lots of test failures that are not augmented for Raft at all, and many test that are augmented are failing for both ZK and Raft cases.

[cmccabe]

Good work spotting the missing @cluster annotations! That is quite a performance win. I wonder if we should disallow tests without this annotation in the future.

[rondagostino]

I wonder if we should disallow tests without [the @cluster] annotation in the future.

I think perhaps yes. Is there a need to allow it? A simple oversight generates a significant parallelism hit at this point -- and they compound quickly as evidenced here.

[cmccabe]

LGTM. Thanks, @rondagostino !