[CI] Speed up CI test and fix flaky test by yzang2019 · Pull Request #1925 · streamnative/kop

yzang2019 · 2023-06-26T07:54:51Z

Motivation

KoP test is too slow, and there are couple of test very flaky and always timeout.

Modifications

Parallelize KoP uni tests with multiple jobs after regroup the tests into corresponding packages
Add a testing matrix for github workflow to spin up one job for each group of tests
Increase timeout to fix some flaky tests

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Check the box below.

Need to update docs?

doc-required

(If you need help on updating docs, create a doc issue)
no-need-doc

(Please explain why)
doc

(If this PR contains doc changes)

CLAassistant · 2023-06-26T07:55:03Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

github-actions · 2023-06-26T07:55:07Z

@yzang2019:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

github-actions · 2023-06-26T07:55:33Z

@yzang2019:Thanks for providing doc info!

yzang2019 · 2023-06-26T07:58:08Z

tests/src/test/java/io/streamnative/pulsar/handlers/kop/TransactionTest.java

    }

-    @Test(timeOut = 1000 * 10, dataProvider = "produceConfigProvider")
+    @Test(timeOut = 1000 * 30, dataProvider = "produceConfigProvider")


I've run these locally and it takes 22 seconds

yzang2019 · 2023-06-26T07:59:32Z

tests/src/test/java/io/streamnative/pulsar/handlers/kop/MetricsProviderTest.java

    }

-    @Test(timeOut = 20000, expectedExceptions = KeeperException.NoNodeException.class)
+    @Test(timeOut = 60000, expectedExceptions = KeeperException.NoNodeException.class)


Tested this locally which takes usually 20-30 seconds to finish

BewareMyPower · 2023-06-27T03:40:26Z

I don't think it speeds up the CI test actually.

You can see the module-test, i.e. the tests under the tests/ directory, took the most of the time of CI (44 minutes), while the other tests only took 14 minutes in summary.

See another CI example in #1914:

How could you say this PR speeds up the CI test?

gaoran10 · 2023-06-27T04:18:10Z

I don't think it speeds up the CI test actually.

Maybe we need to add some test groups for tests and use different ci jobs to test various test groups.

BewareMyPower · 2023-06-27T06:49:35Z

Maybe we need to add some test groups for tests

+1

yzang2019 · 2023-06-28T22:47:56Z

I don't think it speeds up the CI test actually.
You can see the `module-test`, i.e. the tests under the `tests/` directory, took the most of the time of CI (44 minutes), while the other tests only took 14 minutes in summary.
See another CI example in #1914:
How could you say this PR speeds up the CI test?

Yeah, you are right, I only realized this after I split it into multiple jobs, I think next step is to split them into groups

codecov · 2023-07-03T19:54:16Z

Codecov Report

Merging #1925 (b02bd73) into master (f3b95a2) will decrease coverage by 0.70%.
The diff coverage is 3.52%.

@@             Coverage Diff              @@
##             master    #1925      +/-   ##
============================================
- Coverage     18.64%   17.94%   -0.70%     
- Complexity      743      745       +2     
============================================
  Files           186      194       +8     
  Lines         13377    13957     +580     
  Branches       1231     1291      +60     
============================================
+ Hits           2494     2505      +11     
- Misses        10702    11271     +569     
  Partials        181      181

Impacted Files	Coverage Δ
...streamnative/pulsar/handlers/kop/DelayedFetch.java	`0.00% <0.00%> (ø)`
...ative/pulsar/handlers/kop/KafkaCommandDecoder.java	`0.32% <0.00%> (-0.01%)`	⬇️
...tive/pulsar/handlers/kop/KafkaProtocolHandler.java	`0.00% <0.00%> (ø)`
...ative/pulsar/handlers/kop/KafkaRequestHandler.java	`1.07% <0.00%> (-0.01%)`	⬇️
...pulsar/handlers/kop/KafkaTopicConsumerManager.java	`0.00% <0.00%> (ø)`
...e/pulsar/handlers/kop/KafkaTopicLookupService.java	`2.00% <0.00%> (ø)`
...ndlers/kop/coordinator/group/GroupCoordinator.java	`0.00% <0.00%> (ø)`
...rs/kop/coordinator/transaction/PendingRequest.java	`0.00% <0.00%> (ø)`
...kop/coordinator/transaction/TransactionConfig.java	`0.00% <0.00%> (ø)`
...oordinator/transaction/TransactionCoordinator.java	`0.00% <0.00%> (ø)`
... and 23 more

yzang2019 · 2023-07-03T22:17:26Z

@BewareMyPower @Demogorgon314 I've addressed comments, total test time dropped from 50 minutes to around 10 minutes now, could you take another look?

gaoran10

Great work! LGTM, left a trivial comment.

gaoran10 · 2023-07-04T02:39:10Z

tests/src/test/java/io/streamnative/pulsar/handlers/kop/e2e/BasicEndToEndKafkaTest.java


-        @Cleanup
-        final KafkaProducer<String, String> kafkaProducer = newKafkaProducer();
+        @Cleanup final KafkaProducer<String, String> kafkaProducer = newKafkaProducer();


Could we retain the code format?

BewareMyPower

Great work!

### Motivation After streamnative#1925, the surefire artifacts won't be uploaded if tests failed. ### Modifications Upload the surefire artifacts when tests failed.

### Motivation After #1925, the surefire artifacts won't be uploaded if tests failed. ### Modifications Upload the surefire artifacts when tests failed.

### Motivation KoP test is too slow, and there are couple of test very flaky and always timeout. ### Modifications Parallelize KoP uni tests with multiple jobs after regroup the tests into corresponding packages Add a testing matrix for github workflow to spin up one job for each group of tests Increase timeout to fix some flaky tests (cherry picked from commit 24180bb)

Speed up CI test and fix flaky test

61bda9c

yzang2019 requested review from BewareMyPower and Demogorgon314 as code owners June 26, 2023 07:54

github-actions bot assigned yzang2019 Jun 26, 2023

github-actions bot added doc-info-missing This pr needs to mark a document option in description and removed doc-info-missing This pr needs to mark a document option in description labels Jun 26, 2023

github-actions bot added the no-need-doc This pr does not need any document label Jun 26, 2023

BewareMyPower added release/2.11 release/3.0 labels Jun 26, 2023

Fix flaky tests

f7d7b96

yzang2019 commented Jun 26, 2023

View reviewed changes

Add oauth server and fix another flaky test

5b3ed54

Demogorgon314 previously approved these changes Jun 27, 2023

View reviewed changes

yzang2019 changed the title ~~Speed up CI test and fix flaky test~~ [CI] Speed up CI test and fix flaky test Jun 28, 2023

BewareMyPower added cherry-picked/branch-2.11 and removed cherry-picked/branch-2.11 labels Jun 29, 2023

Refactor the structure of tests module

79f82ea

yzang2019 dismissed Demogorgon314’s stale review via 79f82ea July 3, 2023 18:49

yzang2019 added 5 commits July 3, 2023 11:55

Merge branch 'master' into yzang/fix-flaky-test

8e3e724

Fix syntax error

3c67f1c

Fix test names

157806d

test to json

e589c2f

Fix matrix

10c4795

yzang2019 added 7 commits July 3, 2023 12:17

Optimize import

78201cf

Fix import stylei

1d18de8

Reorder imports

753861b

Avoid using * in import

c5cdcd6

Fix line to long issue

f5173b6

Add all tests

2f93f9b

Fix style issue

aa167be

yzang2019 added 2 commits July 3, 2023 15:08

More move arounds

dbf94bd

Fix style

829cdd7

yzang2019 requested review from Demogorgon314 and codelipenghui July 3, 2023 22:17

yzang2019 added 2 commits July 3, 2023 15:42

Fix one more flaky test

9014081

Add unit test check process

b02bd73

gaoran10 reviewed Jul 4, 2023

View reviewed changes

BewareMyPower approved these changes Jul 4, 2023

View reviewed changes

Demogorgon314 approved these changes Jul 4, 2023

View reviewed changes

gaoran10 approved these changes Jul 4, 2023

View reviewed changes

BewareMyPower merged commit 24180bb into streamnative:master Jul 4, 2023

BewareMyPower mentioned this pull request Jul 31, 2023

[CI] Upload surefire artifacts when tests failed #1989

Merged

3 tasks

BewareMyPower added a commit that referenced this pull request Aug 1, 2023

[CI] Upload surefire artifacts when tests failed (#1989)

bdcb731

### Motivation After #1925, the surefire artifacts won't be uploaded if tests failed. ### Modifications Upload the surefire artifacts when tests failed.

Conversation

yzang2019 commented Jun 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Documentation

Uh oh!

CLAassistant commented Jun 26, 2023

Uh oh!

github-actions bot commented Jun 26, 2023

Uh oh!

github-actions bot commented Jun 26, 2023

Uh oh!

yzang2019 Jun 26, 2023

Choose a reason for hiding this comment

Uh oh!

yzang2019 Jun 26, 2023

Choose a reason for hiding this comment

Uh oh!

BewareMyPower commented Jun 27, 2023

Uh oh!

gaoran10 commented Jun 27, 2023

Uh oh!

BewareMyPower commented Jun 27, 2023

Uh oh!

yzang2019 commented Jun 28, 2023

Uh oh!

codecov bot commented Jul 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yzang2019 commented Jul 3, 2023

Uh oh!

gaoran10 left a comment

Choose a reason for hiding this comment

Uh oh!

gaoran10 Jul 4, 2023

Choose a reason for hiding this comment

Uh oh!

BewareMyPower left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yzang2019 commented Jun 26, 2023 •

edited

Loading

codecov bot commented Jul 3, 2023 •

edited

Loading