HDDS-5000. Run CI checks selectively #2479

adoroszlai · 2021-07-30T20:08:48Z

What changes were proposed in this pull request?

Introduce a new job (build-info) in the CI workflow to determine which other jobs should be run. The goal is to skip checks which are not applicable to a specific change.

All tests are run for push and scheduled builds (no change here).
For PRs the incoming (merge) commit's list of changed files is tested against various patterns:
- README changes trigger no tests at all
- rat is run for all other changes
- docs is run for changes in hadoop-hdds/docs
- author is triggered for Java changes
- checkstyle and findbugs are also triggered for POM changes
- dependency is only triggered by POM changes
- acceptance, integration and kubernetes tests are triggered by changes in the corresponding test files
If any uncategorized file is changed (e.g. any non-test Java file), all tests are triggered.
Trigger all tests if specific label (full tests needed) is present in the PR. If it is added later, a new empty commit is needed to re-trigger CI.
Skip all tests for draft PR, same as previously, but without hard-coding in workflow.

Big thanks to Apache Airflow for the libraries and large chunks of the script.

https://issues.apache.org/jira/browse/HDDS-5000

How was this patch tested?

Created test pull requests in my own fork:

Main one with various incremental changes to verify the tests being triggered:
adoroszlai#22

One with full tests needed label, triggers all tests:
adoroszlai#23

Robot test change (minor improvement over the main PR):
adoroszlai#24

Regular CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/1082718710

I have also added a bats-based test suite. It is not run as part of CI currently, because it uses optional bats libraries. I haven't figured out how best to integrate in a developer-friendly way (HDDS-5559). But at least it can be run manually and I found it saves lots of time (not having to open lots of PRs in my fork).

This reverts commit 7a64a7f.

This reverts commit 2c33b7a.

Co-authored-by: GeorgeJahad <github@blackbirdsystems.net>

…into HDDS-5000

GeorgeJahad · 2021-08-09T21:29:01Z

I like the new bats script a lot. A couple of questions:

The new bats script doesn't get invoked by hadoop-ozone/dev-support/checks/bats.sh.
Should it be?
When I ran it, I got these errors:

dev-support/ci/selective_ci_checks.sh: line 174: PR_DRAFT: unbound variable
dev-support/ci/bats-support/load.bash.bash does not exist

which I fixed like so:

-load bats-support/load.bash
-load bats-assert/load.bash
+export PR_DRAFT=false
+load bats-support/load
+load bats-assert/load

Was I running it wrong?

Is it worth adding a comment like this?:


# This script confirms that selective_ci_checks.sh works correctly against known sha's.
# To run it you must clone and then install:
# https://github.com/sstephenson/bats.git

# Then clone the following repos into dev-support/ci:
# git clone https://github.com/ztombol/bats-assert
# git clone https://github.com/ztombol/bats-support

Everything else looks good. I like how the 3 functions were unified.

adoroszlai · 2021-08-09T22:13:48Z

The new bats script doesn't get invoked by hadoop-ozone/dev-support/checks/bats.sh.
Should it be?

Ideally it should, but I know it currently does not. Filed HDDS-5559 for follow-up.

When I ran it, I got these errors:
...
Was I running it wrong?

Thank you for trying it. The only difference is that I used a bit newer version of bats (and its libraries).

Is it worth adding a comment like this?:

Thanks, added with updated URLs.

fapifta · 2021-08-11T13:19:00Z

Hi @adoroszlai

this PR seems to be very useful to reduce our build burden on available resources in github actions. So thank you very much to taking this on.

One question... do we have any general documentation about our CI flow, and how the different parts are working together based on what rules? If not, then it would be a nice follow up to document it together with how this part of the CI system is working in the project, if we already have something it would be great to extend with some docs for new selective checks.

GeorgeJahad · 2021-08-11T15:01:17Z

it would be a nice follow up to document it together with how this
part of the CI system is working in the project, if we already have
something it would be great to extend with some docs for new
selective checks.

@adoroszlai @fapifta I can write something up if you want, but won't have time for a week or so. I would create a separate PR for that. Let me know if that would be helpful.

fapifta · 2021-08-11T15:06:42Z

@GeorgeJahad sounds good to me unless Attila insists on doing it :) Would you mind create a separate JIRA to track the effort, and link it to HDDS-5000 as well?

adoroszlai · 2021-08-11T15:09:51Z

Thanks @GeorgeJahad for the offer. Please feel free to take HDDS-5609.

fapifta

@GeorgeJahad as I believe your questions were addressed just as mine, I am +1 to commit the changes. Please let Attila know if anything is missig still.

vivekratnavel · 2021-08-11T19:04:32Z

Great work @adoroszlai and @GeorgeJahad! I am waiting for this PR to be committed.

GeorgeJahad · 2021-08-11T19:17:11Z

Please let Attila know if anything is missig still.

I've checked all the most recent changes. Everything looks good to me too!
Thank you @adoroszlai !

adoroszlai · 2021-08-11T19:23:19Z

Thanks @fapifta, @GeorgeJahad and @vivekratnavel for the reviews.

* master: HDDS-5358. Incorrect cache entry invalidation causes intermittent failure in testGetS3SecretAndRevokeS3Secret (apache#2518) HDDS-5608. Fix wrong command in ugrade doc (apache#2524) HDDS-5000. Run CI checks selectively (apache#2479) HDDS-4929. Select target datanodes and containers to move for Container Balancer (apache#2441) HDDS-5283. getStorageSize cast to int can cause issue (apache#2303) HDDS-5449 Recon namespace summary 'du' information should return replicated size of a key (apache#2489) HDDS-5558. vUnit invocation unit() may produce NPE (apache#2513) HDDS-5531. For Link Buckets avoid showing metadata. (apache#2502) HDDS-5549. Add 1.1 to supported versions in security policy (apache#2519) HDDS-5555. remove pipeline manager v1 code (apache#2511) HDDS-5546.OM Service ID change causes OM startup failure. (apache#2512) HDDS-5360. DN failed to process all delete block commands in one heartbeat interval (apache#2420) HDDS-5021. dev-support Dockerfile is badly outdated (apache#2480)

adoroszlai added 30 commits July 28, 2021 18:52

HDDS-5000. Run CI checks selectively

4e5f168

Missing needs-dependency-check output

224d6f8

Remove references to undefined OUTPUT_LOG

c35d0b5

TEMP - disable build on push and schedule

2c33b7a

Fix whitespace

9582889

Fetch target branch if needed

50aaa21

Fix fetch command

7a22833

Compare to parent commit instead of CI_TARGET_BRANCH

0dbb56b

Fetch parent

ef1d303

TEMP - keep only build-info job

7a64a7f

Ignore docs/README.md

aef55cf

Simplify ignore logic

4bb0281

Update count_changed_files to match show_changed_files

49c6e44

Unset errexit for greps

5190eb2

Skip env. summary

2048277

Create collapsible group for PR label check

ed689c7

fewer empty lines

6f23722

print results in batch, without grouping

93c8f76

make author conditional, only applies to java files

5cad53a

move output of dependency check to the batch

958ff2c

Remove leftover ignore_array

b724768

move actual output to set_outputs

5d0a4b9

fix output parameters

d903f70

Merge remote-tracking branch 'origin/master' into HDDS-5000

5ad6187

Revert "TEMP - keep only build-info job"

f8a9909

This reverts commit 7a64a7f.

Revert "TEMP - disable build on push and schedule"

b6cebe1

This reverts commit 2c33b7a.

fix quote

367def4

trigger new CI check

802ab55

remove unused var

b473433

let robot files trigger only compose & kubernetes

c417a11

George Jahad and others added 9 commits August 4, 2021 23:12

fixed update_match_files

656bd0c

Add overview comment and reference Airflow source

9c44e08

Co-authored-by: GeorgeJahad <github@blackbirdsystems.net>

Merge remote-tracking branch 'GeorgeJahad/5000-with-list-of-matches' …

254afe3

…into HDDS-5000

reduce code duplication

6248e6e

clarify/simplify file count output

499c82e

remove group for docs check, no output to be enclosed

4f82822

add bats test for selective_ci_checks.sh

db346cf

more tests

c749f1b

Merge remote-tracking branch 'origin/master' into HDDS-5000

f3ba794

adoroszlai added 2 commits August 9, 2021 23:57

document test script

e633280

set default for PR_DRAFT variable

9e8bbe4

trigger new CI check

20f77da

adoroszlai requested a review from vivekratnavel August 10, 2021 07:11

adoroszlai added 4 commits August 10, 2021 15:47

count additional files which should not trigger long-running tests

0a451ae

trigger new CI check

2ded3f5

verify needs-build in test

fc1b2b2

build_needed: unbound variable

46fab85

fapifta approved these changes Aug 11, 2021

View reviewed changes

adoroszlai merged commit ee929dc into apache:master Aug 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-5000. Run CI checks selectively #2479

HDDS-5000. Run CI checks selectively #2479

Uh oh!

adoroszlai commented Jul 30, 2021 •

edited

Loading

Uh oh!

GeorgeJahad commented Aug 9, 2021

Uh oh!

adoroszlai commented Aug 9, 2021

Uh oh!

fapifta commented Aug 11, 2021

Uh oh!

GeorgeJahad commented Aug 11, 2021

Uh oh!

fapifta commented Aug 11, 2021

Uh oh!

adoroszlai commented Aug 11, 2021

Uh oh!

fapifta left a comment

Uh oh!

vivekratnavel commented Aug 11, 2021

Uh oh!

GeorgeJahad commented Aug 11, 2021 •

edited

Loading

Uh oh!

adoroszlai commented Aug 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HDDS-5000. Run CI checks selectively #2479

HDDS-5000. Run CI checks selectively #2479

Uh oh!

Conversation

adoroszlai commented Jul 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

GeorgeJahad commented Aug 9, 2021

Uh oh!

adoroszlai commented Aug 9, 2021

Uh oh!

fapifta commented Aug 11, 2021

Uh oh!

GeorgeJahad commented Aug 11, 2021

Uh oh!

fapifta commented Aug 11, 2021

Uh oh!

adoroszlai commented Aug 11, 2021

Uh oh!

fapifta left a comment

Choose a reason for hiding this comment

Uh oh!

vivekratnavel commented Aug 11, 2021

Uh oh!

GeorgeJahad commented Aug 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adoroszlai commented Aug 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adoroszlai commented Jul 30, 2021 •

edited

Loading

GeorgeJahad commented Aug 11, 2021 •

edited

Loading