Conversation
|
@santhoshct some initial notes:
Thanks for continuing work on this! |
…y links to the test classes and test methods
|
All three successfully done. |
|
This looks excellent! The links will be super helpful. Looks like the "High-Priority Quarantined Tests" table is mostly redundant with the summary table at the top. Actually, with the collapsed details sections, I think we can restructure things a little bit. How about the following layout: I think this will give the report a fairly uniform feel and make it a bit more intuitive to navigate. |
…arantine test report
|
I was also thinking about removing the high priority quarantine test report. the layout looks uniform. I have added it and updated the results in PR description. |
There was a problem hiding this comment.
@santhoshct I ran this locally and copied it here https://gist.github.com/mumrah/b476eb6abe07eb37dd297078e39ebbe0
I'd like to see each flaky method for a given test class (like how the PR description looks). I probably should have been more specific with my suggestion yesterday, but in the tables I think we have to have the test cases in there. The only thing I consider too detailed for the tables is the build scan data.
The new layout looks excellent. Just need to add the test cases back inline to the tables.
edit: btw I also notice since we've hyperlinked the whole class/test name, it makes selecting the text tricky. Let's create a column with a clickable link, e.g. "develocity
|
It was my mistake. I forgot to push the code and just updated the test results in PR description. anyways I have pushed the code now and also modified the hyperlinks. |
|
Thanks @santhoshct! Last request -- can you put the develocity link in a separate column so it aligns? It could be the rightmost column. |
|
Updated with the changes - https://gist.github.com/santhoshct/f801b31c09adc02242c003889ddecf21 |
…e-old-protocol-versions * apache-github/trunk: (25 commits) KAFKA-18270: FindCoordinator v0 incorrectly tagged as deprecated (apache#18262) KAFKA-18284: Add group coordinator records for Streams rebalance protocol (apache#18228) MINOR: Fix flaky state updater test (apache#18253) MINOR: improve StreamsResetter logging (apache#18237) KAFKA-18227: Ensure v2 partitions are not added to last transaction during upgrade (apache#18176) Add IT for share consumer with duration base offet auto reset (apache#18251) KAFKA-18283: Add StreamsGroupDescribe RPC definitions (apache#18230) KAFKA-18241: add docs check to CI (apache#18183) KAFKA-18223 Improve flaky test report (apache#18212) MINOR Remove triage label in nightly job (apache#18147) KAFKA-18294 Remove deprecated SourceTask#commitRecord (apache#18260) KAFKA-18264 Remove NotLeaderForPartitionException (apache#18211) KAFKA-13722: Refactor SerdeGetter (apache#18242) KAFKA-18094 Remove deprecated TopicListing(String, Boolean) (apache#18248) KAFKA-18282: Add StreamsGroupHeartbeat RPC definitions (apache#18227) KAFKA-18026: KIP-1112 migrate KTableSuppressProcessorSupplier (apache#18150) KAFKA-18026: transition KTable#filter impl to use processor wrapper (apache#18205) KAFKA-18293 Remove `org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler` and `org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler` (apache#18244) MINOR: add assertion about groupEpoch and targetAssignmentEpoch to testConsumerGroups (apache#18203) KAFKA-17960; PlaintextAdminIntegrationTest.testConsumerGroups fails with CONSUMER group protocol (apache#18234) ...
Add tables for all sections of the report and add links to Develocity. Reviewers: David Arthur <mumrah@gmail.com>
Testing
Flaky Test Report for 2024-12-17
This report was run on 2024-12-17 08:45:18 UTC
Most Problematic Tests
Found 8 tests that have been quarantined for 7 days and are still failing frequently.
Detailed Execution History
org.apache.kafka.tiered.storage.integration.OffloadAndTxnConsumeFromLeaderTest
executeTieredStorageTest(String, String)[1]
Recent Executions:
executeTieredStorageTest(String, String)[2]
Recent Executions:
kafka.api.CustomQuotaCallbackTest
testExecuteUpdateClusterMetadataMethodCallback(String, String)[1]
Recent Executions:
org.apache.kafka.tiered.storage.integration.TransactionsWithTieredStoreTest
testReadCommittedConsumerShouldNotSeeUndecidedData(String, String)[2]
Recent Executions:
testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[2]
Recent Executions:
testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[1]
Recent Executions:
testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[1]
Recent Executions:
testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[2]
Recent Executions:
org.apache.kafka.common.network.Tls13SelectorTest
testCloseOldestConnection()
Recent Executions:
org.apache.kafka.common.network.SslTransportLayerTest
testSelectorPollReadSize(Args)[1]
Recent Executions:
kafka.api.TransactionsTest
testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[1]
Recent Executions:
testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[2]
Recent Executions:
testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[1]
Recent Executions:
testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[2]
Recent Executions:
kafka.api.PlaintextConsumerTest
testCloseLeavesGroupOnInterrupt(String, String)[2]
Recent Executions:
testCoordinatorFailover(String, String)[2]
Recent Executions:
testCloseLeavesGroupOnInterrupt(String, String)[1]
Recent Executions:
testCoordinatorFailover(String, String)[1]
Recent Executions:
org.apache.kafka.clients.consumer.internals.AbstractCoordinatorTest
testWakeupAfterSyncGroupReceived()
Recent Executions:
Flaky Test Regressions
No flaky test regressions found.
Cleared Tests (Ready for Unquarantine)
Found 20 test classes with 42 test methods that have been consistently passing. These tests could be candidates for removing quarantine annotations at either class or method level.
Detailed Test Method History
kafka.integration.MetricsDuringTopicCreationDeletionTest
testMetricsDuringTopicCreateDelete(String)[1]
Recent Executions:
kafka.api.PlaintextConsumerSubscriptionTest
testRe2JPatternExpandSubscription(String, String)[1]
Recent Executions:
kafka.api.PlaintextAdminIntegrationTest
testConsumerGroupWithMemberMigration(String)[1]
Recent Executions:
kafka.api.SslAdminIntegrationTest
testListNodesFromControllersIncludingFencedBrokers(String)[1]
Recent Executions:
org.apache.kafka.streams.AutoOffsetResetTest
shouldThrowExceptionOnDurationForEarliestReset()
Recent Executions:
shouldThrowExceptionOnDurationForLastetReset()
Recent Executions:
shouldThrowExceptionOnDurationForNoneReset()
Recent Executions:
org.apache.kafka.streams.kstream.internals.KStreamImplTest
shouldProcessWithProcessorAndState()
Recent Executions:
org.apache.kafka.clients.producer.internals.TransactionManagerTest
testTransactionV2AddPartitionAndOffsets()
Recent Executions:
testMaybeAddPartitionToTransactionInTransactionV2()
Recent Executions:
testFatalErrorWhenProduceResponseWithInvalidPidMapping()
Recent Executions:
kafka.api.SslConsumerTest
testCoordinatorFailover(String, String)[1]
Recent Executions:
testCoordinatorFailover(String, String)[2]
Recent Executions:
org.apache.kafka.streams.StreamsBuilderTest
shouldWrapProcessorsForStreamStreamSelfJoinWithSharedStoreOptimization()
Recent Executions:
kafka.api.SaslSslConsumerTest
testCoordinatorFailover(String, String)[1]
Recent Executions:
testCoordinatorFailover(String, String)[2]
Recent Executions:
org.apache.kafka.clients.consumer.internals.ApplicationEventHandlerTest
testRecordApplicationEventQueueSize()
Recent Executions:
org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorderTest
shouldThrowIfMetricRecorderIsInitialisedWithNullTaskId()
Recent Executions:
shouldThrowIfMetricRecorderIsInitialisedWithNullMetrics()
Recent Executions:
org.apache.kafka.controller.QuorumControllerTest
testUncleanShutdownElrDisabled()
Recent Executions:
testUncleanShutdownBrokerElrEnabled()
Recent Executions:
kafka.test.api.ShareConsumerTest
testAcknowledgementCommitCallbackCallsShareConsumerWakeup(String)[1]
Recent Executions:
testAcknowledgementCommitCallbackCallsShareConsumerWakeup(String)[2]
Recent Executions:
testAcknowledgementCommitCallbackInvalidRecordStateException(String)[1]
Recent Executions:
testMultipleConsumersInGroupConcurrentConsumption(String)[1]
Recent Executions:
testExplicitAcknowledgementCommitAsync(String)[1]
Recent Executions:
org.apache.kafka.streams.integration.FineGrainedAutoResetIntegrationTest
shouldResetByDuration()
Recent Executions:
shouldFailForResetNone()
Recent Executions:
kafka.api.SaslMultiMechanismConsumerTest
testCoordinatorFailover(String, String)[1]
Recent Executions:
testCoordinatorFailover(String, String)[2]
Recent Executions:
org.apache.kafka.connect.integration.OffsetsApiIntegrationTest
testResetSinkConnectorOffsetsOverriddenConsumerGroupId()
Recent Executions:
testResetSinkConnectorOffsets()
Recent Executions:
testAlterSinkConnectorOffsetsOverriddenConsumerGroupId()
Recent Executions:
testGetSinkConnectorOffsetsDifferentKafkaClusterTargeted()
Recent Executions:
testAlterSinkConnectorOffsetsDifferentKafkaClusterTargeted()
Recent Executions:
kafka.network.DynamicConnectionQuotaTest
testDynamicConnectionQuota(String)[1]
Recent Executions:
kafka.api.SaslPlaintextConsumerTest
testCoordinatorFailover(String, String)[1]
Recent Executions:
testCoordinatorFailover(String, String)[2]
Recent Executions:
kafka.api.PlaintextConsumerTest
testCoordinatorFailover(String, String)[2]
Recent Executions:
testCoordinatorFailover(String, String)[1]
Recent Executions:
testCloseLeavesGroupOnInterrupt(String, String)[1]
Recent Executions:
testCloseLeavesGroupOnInterrupt(String, String)[2]
Recent Executions:
Committer Checklist (excluded from commit message)