[ci] Split Jenkinsfile into platform-specific jobs #13300

driazati · 2022-11-04T23:10:27Z

This breaks up the Jenkinsfile into ones for GPU, CPU, etc. This removes a false dependency between the build and test steps (e.g. before the GPU tests had to wait on the Hexagon build to complete) and makes the Jenkins UI a bit better since there's not 30 tests to scroll through to find a failure. An example can be found in my fork here: https://github.com/driazati/tvm/pull/38 in the checks box. Before this is merged https://github.com/tlc-pack/ci/blob/main/jenkins/jenkins-jobs/prod/tvm.yaml will need to be updated to accept webhooks from apache/tvm instead of my fork.

See #13337 for more context

tvm-bot · 2022-11-04T23:10:30Z

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

Built docs for commit 90ce205 can be found here.
This PR significantly changed CI runtime: +36.44% (167.82m -> 228.98m)
cc @Mousius, @areusch, @gigiblender, @leandron _{See #10317 for details}

_{Generated by tvm-bot}

mehrdadh · 2022-11-07T16:38:13Z

This is great!!

quic-sanirudh · 2022-11-07T16:46:19Z

This is amazing. It'll make it so much easier to find failures, thanks a lot for this.

mehrdadh · 2022-11-08T17:29:18Z

Another great point about this PR is that now we can easily replicate only a part of the CI in a separate environment. For example, if someone wants to only run Hexagon tests or microTVM tests on physical devices, they could reuse the same Jenkins file for cortexm or hexagon.

ci/jenkins/generate.py

gigiblender · 2022-11-11T15:56:13Z

ci/jenkins/generate.py

+    args = parser.parse_args()
+
+    sources = TEMPLATES_DIR.glob("*jenkinsfile.groovy.j2")
+    changes = [update_jenkinsfile(source) for source in sources if source.name != "base.groovy.j2"]


Not sure if the if condition is necessary

its just there to keep from copying base.groovy.j2 to the generated directory since it doesn't do anything outside of provide helper functions

ci/jenkins/generate.py

gigiblender · 2022-11-11T21:05:35Z

ci/jenkins/generated/arm_jenkinsfile.groovy

+        )
+        skip_ci = should_skip_ci(env.CHANGE_ID)
+        skip_slow_tests = should_skip_slow_tests(env.CHANGE_ID)
+        rebuild_docker_images = sh (


I can see rebuild_docker_images is set and never used in all the generated Jenkins files except the docker one. I don't think this is a particular issue at the moment. Even though it would be nice to have only the current image name used (ci_arm in this case).

However, it makes me wonder if having a separate job for building the docker images allows for a race between a job trying to pull a docker image and the docker_jenkisfile building a new docker image.

Should we maybe have instead each platform (arm, cpu, gpu ...) build its own image if needed?

Right now the image build is just to check the build so it doesn't actually use them later on. We should definitely come in and try to clean this up some more later on since it's still a bit tangled up with the old monolithic job

ci/jenkins/generated/arm_jenkinsfile.groovy

gigiblender · 2022-11-11T21:45:16Z

ci/jenkins/generated/docker_jenkinsfile.groovy

+  )
+}
+
+stage('Docker Image Build') {


This seems to be outside of a method. Are we ok with that?

As long as Jenkins doesn't blow up when running it's ok (the JVM has a 64k limit on method size or something so if there's too much generated code things need to be split up)

ci/jenkins/generated/docker_jenkinsfile.groovy

ci/jenkins/generated/lint_jenkinsfile.groovy

gigiblender · 2022-11-11T22:06:46Z

ci/jenkins/templates/utils/base.groovy.j2


 cancel_previous_build()

 prepare()


We could add a newline here

The EOF-adjacent newlines get caught by https://github.com/apache/tvm/blob/main/tests/lint/whitespace.sh, so we can't really change them without adding an exception

areusch

thanks @driazati, couple questions but i agree with splitting these up

ci/jenkins/data.py

docker/dev_common.sh

ci/scripts/jenkins/open_docker_update_pr.py

ci/jenkins/templates/utils/Build.groovy.j2

ci/jenkins/generated/arm_jenkinsfile.groovy

areusch

thanks @driazati a couple small last things

ci/jenkins/templates/arm_jenkinsfile.groovy.j2

docker/dev_common.sh

areusch

thanks @driazati !

shingjan · 2022-12-06T22:14:38Z

thanks @driazati ! Great work!

This is a follow on to tlc-pack/ci#58 and #13300. This removes the old `Jenkinsfile` and makes the new job statuses required for PRs to merge.

The PR #13300 had a bad merge with main an undid a fix from #13442, this adds it back in.

alter-xp · 2023-01-18T08:59:09Z

hi @driazati , It seems that the contents related to how to regenerate Jenkinsfile in ci/README.md and ci/jenkins/README.md are already out of date. Is there any new tutorial?

driazati · 2023-01-18T11:02:21Z

hi @driazati , It seems that the contents related to how to regenerate Jenkinsfile in ci/README.md and ci/jenkins/README.md are already out of date. Is there any new tutorial?

Ah good catch, I will follow up with a PR to update the docs but you can run this to re-sync everything

python3 -m pip install jinja2
python ci/jenkins/generate.py

The PR apache#13300 had a bad merge with main an undid a fix from apache#13442, this adds it back in.

This breaks up the Jenkinsfile into ones for GPU, CPU, etc. This removes a false dependency between the build and test steps (e.g. before the GPU tests had to wait on the Hexagon build to complete) and makes the Jenkins UI a bit better since there's not 30 tests to scroll through to find a failure. An example can be found in my fork here: https://github.com/driazati/tvm/pull/38 in the checks box. Before this is merged https://github.com/tlc-pack/ci/blob/main/jenkins/jenkins-jobs/prod/tvm.yaml will need to be updated to accept webhooks from apache/tvm instead of my fork. See apache#13337 for more context

…he#13316) This is a follow on to tlc-pack/ci#58 and apache#13300. This removes the old `Jenkinsfile` and makes the new job statuses required for PRs to merge.

The PR apache#13300 had a bad merge with main an undid a fix from apache#13442, this adds it back in.

driazati marked this pull request as ready for review November 7, 2022 17:35

driazati mentioned this pull request Nov 7, 2022

[ci] Remove Jenkinsfile for migration to platform-specific jobs #13316

Merged

driazati mentioned this pull request Nov 9, 2022

[ci] Split up Jenkins into platform-specific jobs #13337

Closed

8 tasks

gigiblender reviewed Nov 11, 2022

View reviewed changes

driazati requested review from areusch and gigiblender and removed request for gigiblender November 18, 2022 22:26

areusch reviewed Dec 1, 2022

View reviewed changes

areusch reviewed Dec 2, 2022

View reviewed changes

ci/jenkins/templates/arm_jenkinsfile.groovy.j2 Show resolved Hide resolved

docker/dev_common.sh Outdated Show resolved Hide resolved

driazati requested a review from areusch December 2, 2022 23:31

driazati added 5 commits December 5, 2022 14:03

rebase

a694ad1

Rebase

2a90431

Comments

6810485

Fix lint

c31aba8

Fix tests

90ce205

areusch approved these changes Dec 6, 2022

View reviewed changes

driazati merged commit 9374738 into apache:main Dec 6, 2022

driazati added a commit that referenced this pull request Dec 7, 2022

[ci] Remove Jenkinsfile for migration to platform-specific jobs (#13316)

1a99d75

This is a follow on to tlc-pack/ci#58 and #13300. This removes the old `Jenkinsfile` and makes the new job statuses required for PRs to merge.

driazati mentioned this pull request Dec 7, 2022

[ci] Fix docs deploy #13570

Merged

Mousius pushed a commit that referenced this pull request Dec 7, 2022

[ci] Fix docs deploy (#13570)

3745ccf

The PR #13300 had a bad merge with main an undid a fix from #13442, this adds it back in.

leandron mentioned this pull request Feb 1, 2023

TVM v0.11.0 Release Candidate Notes #13899

Closed

fzi-peccia pushed a commit to fzi-peccia/tvm that referenced this pull request Mar 27, 2023

[ci] Fix docs deploy (apache#13570)

ec4f585

The PR apache#13300 had a bad merge with main an undid a fix from apache#13442, this adds it back in.

mikeseven pushed a commit to mikeseven/tvm that referenced this pull request Sep 27, 2023

[ci] Fix docs deploy (apache#13570)

60cb78f

The PR apache#13300 had a bad merge with main an undid a fix from apache#13442, this adds it back in.

[ci] Split Jenkinsfile into platform-specific jobs #13300

[ci] Split Jenkinsfile into platform-specific jobs #13300

Uh oh!

Conversation

driazati commented Nov 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tvm-bot commented Nov 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mehrdadh commented Nov 7, 2022

Uh oh!

quic-sanirudh commented Nov 7, 2022

Uh oh!

mehrdadh commented Nov 8, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

areusch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

areusch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

areusch left a comment

Choose a reason for hiding this comment

Uh oh!

shingjan commented Dec 6, 2022

Uh oh!

alter-xp commented Jan 18, 2023

Uh oh!

driazati commented Jan 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

driazati commented Nov 4, 2022 •

edited

Loading

tvm-bot commented Nov 4, 2022 •

edited

Loading