Overhaul spack CI #89

BenjaminRodenberg · 2021-03-03T20:04:59Z

This PR simplifies the spack CI.

Uses approach proposed in haampie-spack/ci-example-docker@b98a5e8.
Uses Cron job for creating base image
Automatically gets dependencies from package.py using --only=dependencies
Using jinja and template is not required anymore, since we now use dev-build to build the package directly from the branch

Thanks again @haampie for your talk at SIAM CSE 21! Feel free to add comments or a review to this PR, if you want.

Seems to be missing in precice spack package. See https://github.com/precice/python-bindings/runs/2014465956?check_suite_focus=true#step:8:734

…dings into spack-ci-improvements

BenjaminRodenberg · 2021-03-03T20:09:43Z

74a5dbf might break the tests, but this should be resolved as soon as the upstream PR spack/spack#22059 arrives here.

fsimonis · 2021-03-04T11:30:51Z

Why do we need the local py-pyprecice spack package? Isn't this enough for this pipeline?

haampie · 2021-03-04T11:38:21Z

I usually add add a copy of package.py to the sources so that I can change the build script, add dependencies, etc in a pull request and test it immediately. When merged I create a PR to upstream spack with the new package.py. The downside is that you manually have to sync it with upstream changes to your package.

ajaust · 2021-03-04T13:22:15Z

@fsimonis I think the idea was to have a CI pipeline that one can run on any branch and not just develop..

I personally like(d) the approach using a package.py such that one can set up a proper environment with root spec. I can see that the (old) approach using the template to generate a package.py that references the branch you actually want to test had its advantages. However, our workflow has not been optimal as we also build a new package.py more or less by hand for merging it into the Spack repository. Maybe it would have been worth extending the template approach to generate new package.py for ci and for updating the actual Spack package?

I am no expert on Spack's dev-build feature. If it makes life much easier it could be worth it after all. However, I also read some discussion about either the dev-build or build-dev command of Spack where it was not clear whether the command will be removed at some point or not. I am not sure what the status on that is at the moment. I could not find the discussion either.

ajaust

i added my reviews. I think its basically the question whether dev-build is better than using the template-generated package.py.

.github/workflows/build-spack.yml

ajaust · 2021-03-04T13:25:26Z

spack/ci-spack-pyprecice-deps-1804.dockerfile

+RUN spack --color=always env create --without-view ci /sources/spack/spack.yaml && \
+    spack --color=always -e ci repo add /sources/spack/repo && \
+    spack --color=always -e ci install --fail-fast --only=dependencies && \
+    spack --color=always clean -a


I think we have to evaluate here what is better. The approach have specs in the environment or the new one. Why do you create the environment ci using --without-view ci?

I copied this approach from https://github.com/haampie-spack/ci-example/blob/b98a5e8b1fb7e8aec67707a7eced78efa92b0619/tools/docker/build-env.Dockerfile#L7-L12. So I cannot really give a good reason. I'm not sure, if this is really a satisfactory answer, but I'll try to get through the review with it ;)

spack/repo/packages/py-pyprecice/package.py

ajaust · 2021-03-04T13:28:20Z

spack/spack.yaml

@@ -0,0 +1,6 @@
+spack:
+  specs:
+    - py-pyprecice@develop


Is there a risk that it will be misinterpreted because @develop might not point to the actual develop branch?

I think for our testing this "misinterpretation" is exactly what we want: dev-build uses the current version of an arbitrary branch as a patch for py-pyprecice@develop. Then we install/use (the patched) py-pyprecice@develop in all our tests. Does this make sense? If yes: @ajaust feel free to mark this as resolved. If no: Let's try to get to a conclusion here. I should probably also document this somewhere...

I removed the spack.yaml in e3371fd. I think this new approach is clearer (I hope it works as expected).

BenjaminRodenberg · 2021-03-04T13:37:58Z

@fsimonis I think the idea was to have a CI pipeline that one can run on any branch and not just develop..

Exactly. I think this is a nice feature, since it allows us to test compatibility of a feature with spack before we merge it into develop.

If we take this feature as a "must", then I really prefer the dev-build over the jinja template solution for the following reason:

I think it is more simple (of course subjective)
it allows to use the same package.py for submitting to the upstream spack repository and testing
I was in the end to remove quite some documentation here
I could also remove a lot of code required by jinja in the gitlab action workflows

BenjaminRodenberg · 2021-03-04T17:21:54Z

Currently the test fails due to changes in bad16b0. As soon as this job is done, I will rerun the test and it is expected to succeed.

BenjaminRodenberg · 2021-03-04T21:08:08Z

Currently the test fails, because openssh is missing on the baseimage. This job is currently rebuilding the baseimage including openssh, but this will take a bit longer.

This reverts commit 14d8a7f.

fsimonis · 2021-03-08T14:00:35Z

I think the idea was to have a CI pipeline that one can run on any branch and not just develop.

With dev-build you can run the pipeline on every branch as develop: "Dear spack, please install this package at the given spec, but use the sources in this directory instead of downloading them."

.github/workflows/build-spack.yml

ajaust

Looks good to me know. I assume the dev-build discussion is (somewhat) resolved for the moment.

haampie · 2021-03-12T15:26:13Z

Fwiw, you might also be interested in spack/spack#22115

BenjaminRodenberg added 27 commits February 25, 2021 23:14

Use super.

e25c969

Remove incorrect and update correct file.

e626b71

Experiment with architecture proposed in https://github.com/haampie-s…

392dd78

…pack/ci-example.

Add workflow for building dependencies from https://github.com/haampi…

d6d1d27

…e-spack/ci-example.

Rename

97c513d

build-env.yml

a88bf47

Fix paths and names.

1289eb5

Manually add pkgconfig

cea094c

Seems to be missing in precice spack package. See https://github.com/precice/python-bindings/runs/2014465956?check_suite_focus=true#step:8:734

Remove unneeded file.

63b2248

Merge branch 'spack-ci-improvements' of github.com:precice/python-bin…

73faeff

…dings into spack-ci-improvements

Add hotfix for pkgconfig.

c34d984

Don't install pkgconfig manually

a898c2e

Use spack dev-build, don't use jinja.

c891c6d

Use py-pyprecice@devleop

3a6b3ff

Increase timeout.

aee5bc8

Fix name of env

3c5e2e4

Don't remove environment

b2c861f

Fix package.py

6d890ec

Try overwriting package.py

9691800

Only build env on a weekly basis.

ab4d3a9

Fix env name.

00c17b9

Comment out install for older versions

d4ff597

Go to independent directory to avoid false imports

69fdaac

Explicitly load py-pyprecice

186bd36

Load all dependencies.

aad7ecd

Revert changes in install.

838c18a

Update README.md

1048f8d

BenjaminRodenberg added the enhancement New feature or request label Mar 3, 2021

BenjaminRodenberg requested review from ajaust and fsimonis March 3, 2021 20:04

ajaust reviewed Mar 4, 2021

View reviewed changes

BenjaminRodenberg added 3 commits March 4, 2021 15:38

Merge branch 'develop' into spack-ci-improvements

8ee7c76

Remove spack.yaml.

e3371fd

Only add repo, not all sources.

bad16b0

BenjaminRodenberg added 4 commits March 4, 2021 21:14

Remove WORKDIR.

1a61b7e

Merge branch 'develop' into spack-ci-improvements

6eca69a

Merge branch 'develop' into spack-ci-improvements

da23d16

Remove py-wheel.

5b08b87

BenjaminRodenberg mentioned this pull request Mar 5, 2021

build_spack sporadically fails due to timeout #81

Closed

BenjaminRodenberg added 4 commits March 7, 2021 19:55

Copy is not necessary due to dev-build.

14d8a7f

Revert "Copy is not necessary due to dev-build."

b7e4d5d

This reverts commit 14d8a7f.

Print arch and modules

00b9101

Build for generic architecture x86_64. #81

6b37ff6

Specify target.

17dabd8

BenjaminRodenberg commented Mar 9, 2021

View reviewed changes

.github/workflows/build-spack.yml Show resolved Hide resolved

BenjaminRodenberg requested a review from ajaust March 9, 2021 11:26

BenjaminRodenberg linked an issue Mar 9, 2021 that may be closed by this pull request

build_spack sporadically fails due to timeout #81

Closed

ajaust approved these changes Mar 10, 2021

View reviewed changes

BenjaminRodenberg merged commit 5fed8bd into develop Mar 23, 2021

BenjaminRodenberg deleted the spack-ci-improvements branch March 23, 2021 17:44

IshaanDesai mentioned this pull request Apr 7, 2021

Release v2.2.0.2 #96

Merged

Uh oh!

Overhaul spack CI #89

Overhaul spack CI #89

Uh oh!

Conversation

BenjaminRodenberg commented Mar 3, 2021

Uh oh!

BenjaminRodenberg commented Mar 3, 2021

Uh oh!

fsimonis commented Mar 4, 2021

Uh oh!

haampie commented Mar 4, 2021

Uh oh!

ajaust commented Mar 4, 2021

Uh oh!

ajaust left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ajaust Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

BenjaminRodenberg Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ajaust Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

BenjaminRodenberg Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

BenjaminRodenberg Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

BenjaminRodenberg commented Mar 4, 2021

Uh oh!

BenjaminRodenberg commented Mar 4, 2021

Uh oh!

BenjaminRodenberg commented Mar 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fsimonis commented Mar 8, 2021

Uh oh!

Uh oh!

ajaust left a comment

Choose a reason for hiding this comment

Uh oh!

haampie commented Mar 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

BenjaminRodenberg commented Mar 4, 2021 •

edited

Loading