add support for letting the bot build compat layer by bedroge · Pull Request #163 · EESSI/compatibility-layer

bedroge · 2023-01-06T15:55:35Z

The new script install-compatibility-layer.sh will run Ansible inside an Apptainer container (generated with a modified version of the existing Dockerfile, updated to Rocky 8).

Haven't managed to do a full test run, due to issues with the bootstrapping process: it fails to fetch some source file, so I'll probably have to update the gentoo commit that we use.

bedroge · 2023-01-12T14:09:15Z

This seems to work now. I've tested it, but had to disable the package set installation (as, for instance, archspec wouldn't install due to changes in gentoo, unrelated to this PR) and use a more up-to-date version of the bootstrap script and gentoo commit.

@@ -0,0 +1,17 @@
+FROM debian:11-slim


+RUN apt-get update
+RUN apt-get install -y gcc g++ make diffutils libgmp-dev perl
+RUN apt-get install -y git python3-pip
+RUN pip3 install --upgrade pip


bedroge · 2023-01-12T14:52:20Z

Initially, I switched from CentOS to Rocky for the container, but Rocky 8 only provides Docker containers for arm64 and amd64. So, in the end, I switched to Debian 11, which supports a whole range of architectures including ppc64le (and the unstable Debian image already supports riscv64 as well).

ocaisa · 2023-01-12T15:00:12Z

See https://github.com/EESSI/compatibility-layer/actions/runs/3903315061/jobs/6667515735#step:7:3652 for the failure...no rust compiler for POWER?

ocaisa · 2023-01-12T15:03:10Z

almalinux:8 does have a container for POWER

bedroge · 2023-01-12T15:06:41Z

almalinux:8 does have a container for POWER

Yeah, I considered that one as well, but since Debian is a bit more active with respect to riscv64 as well (which we probably want/need in the future) and the distro shouldn't really matter, I preferred it over Alma.

bedroge · 2023-01-12T15:10:37Z

See https://github.com/EESSI/compatibility-layer/actions/runs/3903315061/jobs/6667515735#step:7:3652 for the failure...no rust compiler for POWER?

Weird, I think it should exist: https://packages.debian.org/bullseye/rustc

bedroge · 2023-03-06T15:06:06Z

Successfully ran this on x86_64:

TASK [compatibility_layer : Run ReFrame tests] *********************************
ok: [localhost]

RUNNING HANDLER [compatibility_layer : Generate locales] ***********************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=57   changed=31   unreachable=0    failed=0    skipped=3    rescued=0    ignored=1

bedroge · 2023-03-07T14:16:12Z

This PR includes so many different things now, I'm splitting it up into smaller ones:

Update to 2023.03 #166 -> updates required for 2023.03 compat layer
Run the entire installation of the compat layer in a container, without requiring root privileges #167 -> functionality for running the full installation in a container
Install ReFrame in a virtual environment if it's not available #168 -> install ReFrame in a venv
Add a cleanup task to the Ansible role #169 -> add the cleanup step

boegel · 2023-03-14T08:50:35Z

+host_arch=$(uname -m)
+eessi_arch=${cpu_target_arch:-${host_arch}}
+eessi_os=linux
+eessi_version=2023.02


bump to 2023.03?

version bump is done or needs to be done in a follow up PR

trz42

All conflicts resolved. Some requested changes (version bump) postponed to follow-up PR.

bedroge added 11 commits January 6, 2023 16:02

install ansible and deps for playbook

b75a299

remove singularity from prefix installation command

5df6a9e

install on localhost, don't use sudo

daa06ea

remove stuff assuming this is run as root

4b35bc9

rename Dockerfile for rocky

06b0b91

use rockylinux8 dockerfile instead of centos8

bbe339c

script for installing the compat layer in a container

c497aa0

run git and ansible as separate steps

ab901bb

remove centos8 dockerfile

a7ef4e9

clean up

39aab5c

remove line used for testing

30f3670

github-advanced-security AI found potential problems Jan 6, 2023

View reviewed changes

Comment thread Dockerfile.bootstrap-prefix-rockylinux8 Fixed

Comment thread Dockerfile.bootstrap-prefix-rockylinux8 Fixed

Comment thread Dockerfile.bootstrap-prefix-rockylinux8 Fixed

bedroge added 7 commits January 12, 2023 11:24

no need to find username

90455ee

don't create portage user+group

026cfc1

pip install reframe if it's not available

6f9c65d

run all commands in one container session, unset LD_LIBRARY_PATH

91ae437

add portage user/group

62f6912

change yes to true

8c2fdbb

add changed_when: false, the command is only used for checking something

408fee4

bedroge marked this pull request as ready for review January 12, 2023 14:09

make comment more clear

8b9ef7d

github-advanced-security AI found potential problems Jan 12, 2023

View reviewed changes

bedroge force-pushed the ansible_in_container branch from 83619d2 to 8b9ef7d Compare January 12, 2023 14:44

use debian11 instead of rockylinux8

6341d27

bedroge added 14 commits February 24, 2023 17:40

use most recent gentoo commit

31f72e1

use first gcc in the list as default

36e1a6f

updates for new compat layer

0dbc4fe

add list of packages to be removed after bootstrap

2ec7714

add prefix_unmask_packages

a0ba29e

add task for unmasking packages

941f406

include the cleanup task

5d06a4c

add reframe_venv_dir var

b9bd86c

use venv for reframe installation

cdc6ee9

add task for removing redundant packages

380093a

use latest bootstrap script

7202e1b

use latest gentoo commit

b15ea7d

restore command for setting pipefail

92e1e67

make script executable

f78f997

bedroge mentioned this pull request Mar 6, 2023

Add direnv to compat layer #159

Closed

bedroge added 2 commits March 7, 2023 10:30

bump version to 2023.03

2b89e8e

use cryptography from os

1f938c5

bedroge changed the title ~~Run the entire compatibility layer installation in a container + add support for letting the bot build~~ Run the entire compatibility layer installation in a container + add support for letting the bot build + updates for 2023.03 Mar 7, 2023

boegel previously requested changes Mar 14, 2023

View reviewed changes

trz42 pushed a commit to trz42/compatibility-layer that referenced this pull request Mar 30, 2023

add bot/build.sh from PR EESSI#163

2b49a58

This was referenced Mar 30, 2023

new compatibility layer (spring 2023) NorESSI/compatibility-layer#1

Closed

build compatibility layer with the bot #173

Closed

Merge branch 'main' into ansible_in_container

e4d91d9

trz42 approved these changes Apr 3, 2023

View reviewed changes

trz42 merged commit b71060f into EESSI:main Apr 3, 2023

boegel changed the title ~~Run the entire compatibility layer installation in a container + add support for letting the bot build + updates for 2023.03~~ add support for letting the bot build compat layer Apr 11, 2023

bedroge deleted the ansible_in_container branch April 21, 2023 13:40

Conversation

bedroge commented Jan 6, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedroge commented Jan 12, 2023

Uh oh!

Check warning

Check warning

Uh oh!

bedroge commented Jan 12, 2023

Uh oh!

ocaisa commented Jan 12, 2023

Uh oh!

ocaisa commented Jan 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedroge commented Jan 12, 2023

Uh oh!

bedroge commented Jan 12, 2023

Uh oh!

bedroge commented Mar 6, 2023

Uh oh!

bedroge commented Mar 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel Mar 14, 2023

Choose a reason for hiding this comment

Uh oh!

trz42 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ocaisa commented Jan 12, 2023 •

edited

Loading

bedroge commented Mar 7, 2023 •

edited

Loading