test installing EasyBuild v3.8.2 to `software.eessi.io/versions/2023.06` by boegel · Pull Request #29 · boegel/software-layer

boegel · 2023-11-25T14:42:31Z

Using this for testing:

stack of PRs for EESSI/software-layer, see overview in:
- update script for building for software.eessi.io (version 2023.06) EESSI/software-layer#383);
enhancement for bot to have repo-specific bucket:
- add support for specifying bucket name as mapping with repo id as key EESSI/eessi-bot-software-layer#230
- https://github.com/EESSI/bot-configs/pull/6

Allow fix incorrect generic path

remove bot configuration (main)

…oftware.eessi.io/versions/2023.06

…d v4.8.2 installations

…2023.06-software.eessi.io_TEST_BOEGEL

…e.eessi.io_TEST_BOEGEL

…-software.eessi.io_TEST_BOEGEL

…o 2023.06-software.eessi.io_TEST_BOEGEL

test-eessi-bot-software-layer · 2023-11-25T14:42:35Z

Instance eessi-bot-test is configured to build:

arch x86_64/generic for repo EESSI-pilot
arch aarch64/generic for repo EESSI-pilot

boegel · 2023-11-25T14:43:17Z

bot: build

test-eessi-bot-software-layer · 2023-11-25T14:43:20Z

Updates by the bot instance eessi-bot-test (click for details)

received bot command build from boegel
- expanded format: build

boegel · 2023-11-25T14:49:03Z

bot: build

test-eessi-bot-software-layer · 2023-11-25T14:49:07Z

Updates by the bot instance eessi-bot-test (click for details)

received bot command build from boegel
- expanded format: build
handling command build resulted in:
- submitted job 1212, for details & status see test installing EasyBuild v3.8.2 to software.eessi.io/versions/2023.06 #29 (comment)
- submitted job 1213, for details & status see test installing EasyBuild v3.8.2 to software.eessi.io/versions/2023.06 #29 (comment)

test-eessi-bot-software-layer · 2023-11-25T14:49:12Z

New job on instance eessi-bot-test for architecture x86_64-generic for repository eessi.io-2023.06-software in job dir /home/boegel/eessi-bot-software-layer/jobs/2023.11/pr_29/1212

date	job status	comment
Nov 25 14:49:11 UTC 2023	submitted	job id `1212` awaits release by job manager
Nov 25 14:49:51 UTC 2023	released	job awaits launch by Slurm scheduler
Nov 25 14:55:56 UTC 2023	running	job `1212` is running
Nov 25 15:03:09 UTC 2023	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-1212.out` ❌ found message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts Details No artefacts were created or found.

test-eessi-bot-software-layer · 2023-11-25T14:49:13Z

New job on instance eessi-bot-test for architecture aarch64-generic for repository eessi.io-2023.06-software in job dir /home/boegel/eessi-bot-software-layer/jobs/2023.11/pr_29/1213

date	job status	comment
Nov 25 14:49:12 UTC 2023	submitted	job id `1213` awaits release by job manager
Nov 25 14:49:49 UTC 2023	released	job awaits launch by Slurm scheduler
Nov 25 14:54:53 UTC 2023	running	job `1213` is running
Nov 25 15:03:07 UTC 2023	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-1213.out` ❌ found message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts Details No artefacts were created or found.

boegel · 2023-11-25T15:06:28Z

Failed with:

ERROR: Values for EESSI_SOFTWARE_SUBDIR_OVERRIDE (aarch64/generic) and EESSI_SOFTWARE_SUBDIR (aarch64/arm/neoverse-n1) differ!

That likely has to do with switching to archdetect instead of using archspec, seems like archdetect doesn't honor $EESSI_SOFTWARE_SUBDIR_OVERRIDE ...

edit: fixed in EESSI#397

Allow `archdetect` to print all possible cpu paths + fix paths for `aarch64/*` (drop `/arm/` subdir)

…pported currently in software.eessi.io

…heck for directories in software-layer if it has been populated already

…6-software.eessi.io_TEST_BOEGEL

boegel · 2023-11-25T15:50:09Z

bot: build

test-eessi-bot-software-layer · 2023-11-25T15:50:11Z

Updates by the bot instance eessi-bot-test (click for details)

received bot command build from boegel
- expanded format: build
handling command build resulted in:
- submitted job 1214, for details & status see test installing EasyBuild v3.8.2 to software.eessi.io/versions/2023.06 #29 (comment)
- submitted job 1215, for details & status see test installing EasyBuild v3.8.2 to software.eessi.io/versions/2023.06 #29 (comment)

test-eessi-bot-software-layer · 2023-11-25T15:50:18Z

New job on instance eessi-bot-test for architecture x86_64-generic for repository eessi.io-2023.06-software in job dir /home/boegel/eessi-bot-software-layer/jobs/2023.11/pr_29/1214

date	job status	comment
Nov 25 15:50:17 UTC 2023	submitted	job id `1214` awaits release by job manager
Nov 25 15:51:18 UTC 2023	released	job awaits launch by Slurm scheduler
Nov 25 15:52:22 UTC 2023	running	job `1214` is running
Nov 25 15:55:28 UTC 2023	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-1214.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-generic-1700927661.tar.gz` size: 19 MiB (19979074 bytes) entries: 31127 modules under 2023.06/software/linux/x86_64/generic/modules/all `EasyBuild/4.8.2.lua` software under 2023.06/software/linux/x86_64/generic/software `EasyBuild/4.8.2` other under 2023.06/software/linux/x86_64/generic `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp` `.lmod/lmodrc.lua`
Nov 25 15:58:15 UTC 2023	not uploaded	transfer of `eessi-2023.06-software-linux-x86_64-generic-1700927661.tar.gz` to S3 bucket failed
Nov 25 16:09:32 UTC 2023	not uploaded	transfer of `eessi-2023.06-software-linux-x86_64-generic-1700927661.tar.gz` to S3 bucket failed
Nov 25 16:16:21 UTC 2023	not uploaded	transfer of `eessi-2023.06-software-linux-x86_64-generic-1700927661.tar.gz` to S3 bucket failed
Nov 25 16:19:07 UTC 2023	not uploaded	transfer of `eessi-2023.06-software-linux-x86_64-generic-1700927661.tar.gz` to S3 bucket failed

test-eessi-bot-software-layer · 2023-11-25T15:50:19Z

New job on instance eessi-bot-test for architecture aarch64-generic for repository eessi.io-2023.06-software in job dir /home/boegel/eessi-bot-software-layer/jobs/2023.11/pr_29/1215

date	job status	comment
Nov 25 15:50:19 UTC 2023	submitted	job id `1215` awaits release by job manager
Nov 25 15:51:17 UTC 2023	released	job awaits launch by Slurm scheduler
Nov 25 15:52:20 UTC 2023	running	job `1215` is running
Nov 25 15:55:27 UTC 2023	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-1215.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-generic-1700927656.tar.gz` size: 19 MiB (19934888 bytes) entries: 31127 modules under 2023.06/software/linux/aarch64/generic/modules/all `EasyBuild/4.8.2.lua` software under 2023.06/software/linux/aarch64/generic/software `EasyBuild/4.8.2` other under 2023.06/software/linux/aarch64/generic `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp` `.lmod/lmodrc.lua`
Nov 25 15:58:17 UTC 2023	not uploaded	transfer of `eessi-2023.06-software-linux-aarch64-generic-1700927656.tar.gz` to S3 bucket failed
Nov 25 16:09:34 UTC 2023	not uploaded	transfer of `eessi-2023.06-software-linux-aarch64-generic-1700927656.tar.gz` to S3 bucket failed
Nov 25 16:16:25 UTC 2023	not uploaded	transfer of `eessi-2023.06-software-linux-aarch64-generic-1700927656.tar.gz` to S3 bucket failed
Nov 25 16:19:39 UTC 2023	not uploaded	transfer of `eessi-2023.06-software-linux-aarch64-generic-1700927656.tar.gz` to S3 bucket failed

boegel · 2023-11-25T15:57:15Z

bot: deploy

test-eessi-bot-software-layer · 2023-11-25T15:57:18Z

Updates by the bot instance eessi-bot-test (click for details)

received bot command deploy from boegel
- expanded format: deploy
handling command deploy failed with message
unknown command deploy; use bot: help for usage information

boegel · 2023-11-25T16:16:35Z

2nd deploy attempt failed because aws command was not available yet in test setup

boegel · 2023-11-25T16:18:23Z

3rd deploy attempt failed because endpoint_url was set to dummy value URL_TO_S3_SERVER in app.cfg

boegel · 2023-11-25T16:22:41Z

last (4th) deploy attempt triggered successfully, only failed due to lack of credentials, which is a problem with the bot environment, not with the code:

[20231125-T16:18:36] Using bucket for eessi.io-2023.06-software: kh-eessi-staging
[20231125-T16:18:36] run_subprocess(): 'Upload tarball to S3 bucket' by running '/home/boegel/eessi-bot-software-layer/scripts/eessi-upload-to-staging --bucket-name kh-eessi-staging --repository boegel/software-layer --pull-request 29 /home/boegel/eessi-bot-software-layer/jobs/2023.11/pr_29/1214/eessi-2023.06-software-linux-x86_64-generic-1700927661.tar.gz' in directory '/home/boegel/eessi-bot-software-layer'
...
           stderr 'upload failed: jobs/2023.11/pr_29/1215/eessi-2023.06-software-linux-aarch64-generic-1700927656.tar.gz to s3://kh-eessi-staging/2023.06/software/linux/aarch64/generic/1700927656/eessi-2023.06-software-linux-aarch64-generic-1700927656.tar.gz Unable to locate credentials

What matters is that it used the current bucket from the following spec in app.cfg

bucket_name = {
    "eessi-hpc.org-2023.06-compat": "eessi-staging-2023.06",
    "eessi-hpc.org-2023.06-software": "eessi-staging-2023.06",
    "eessi.io-2023.06-compat": "software.eessi.io-2023.06",
    "eessi.io-2023.06-software": "kh-eessi-staging" }

ocaisa and others added 24 commits June 16, 2023 14:32

Allow archdetect to print all possible cpu paths

3fc1746

Allow fix incorrect generic path

Add tests

551d008

Remove need to pass argument to function

7db0616

Make sure all the directories actually exist

b81b729

Fix steps in action

739a92d

Fix ARM detection for archdetect

45dc7da

Fix ARM spec

71cf7c9

Also fix archdetect tests

98a95fe

Make tests flexible in terms of EESSI version

ef1a53a

Fix hash of EESSI action

5665fad

Careful with quotes

07bfbb1

Also drop arm subdir from tests

b10415b

Remove EESSI version selection

409ba53

remove bot configuration (main)

92b7649

Merge pull request EESSI#357 from boegel/main_bot_cfg_cleanup

8d8be67

remove bot configuration (main)

update CI workflow for checking for missing installations in /cvmfs/s…

753296f

…oftware.eessi.io/versions/2023.06

get rid of 'pilot' terminology in scripts

a095a9d

update eessi_container.sh script for software.eessi.io

50ec01a

remove easystack files used for pilot.eessi-hpc.org

ad6604f

add initial easystack file for software.eessi.io to check on EasyBuil…

83de8ea

…d v4.8.2 installations

Merge branch '2023.06-software.eessi.io_eessi-container-script' into …

ff873b1

…2023.06-software.eessi.io_TEST_BOEGEL

Merge branch '2023.06-software.eessi.io_scripts' into 2023.06-softwar…

813e7f1

…e.eessi.io_TEST_BOEGEL

Merge branch '2023.06-software.eessi.io_easystack-eb482' into 2023.06…

77969d9

…-software.eessi.io_TEST_BOEGEL

Merge branch '2023.06-software.eessi.io_ci-missing-installations' int…

89a1dbf

…o 2023.06-software.eessi.io_TEST_BOEGEL

boegel added 7 commits November 25, 2023 16:13

Merge pull request EESSI#264 from ocaisa/ocaisa-patch-2

686ac58

Allow `archdetect` to print all possible cpu paths + fix paths for `aarch64/*` (drop `/arm/` subdir)

Merge branch 'main' into 2023.06-software.eessi.io

a4169f6

comment out power9le test for archdetect, since that target is not su…

34adbe5

…pported currently in software.eessi.io

check that archdetect honors $EESSI_SOFTWARE_SUBDIR_OVERRIDE + only c…

468c424

…heck for directories in software-layer if it has been populated already

fix picking up on $EESSI_SOFTWARE_SUBDIR_OVERRIDE in archdetect

04a71f7

mount software.eessi.io when testing archdetect

d0f26f8

Merge branch '2023.06-software.eessi.io_archdetect-fixes' into 2023.0…

d3d9a22

…6-software.eessi.io_TEST_BOEGEL

boegel added bot:deploy and removed bot:deploy labels Nov 25, 2023

boegel closed this Nov 25, 2023

boegel deleted the 2023.06-software.eessi.io_TEST_BOEGEL branch November 25, 2023 16:22

boegel mentioned this pull request Nov 25, 2023

add support for specifying bucket name as mapping with repo id as key EESSI/eessi-bot-software-layer#230

Merged

Conversation

boegel commented Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

test-eessi-bot-software-layer Bot commented Nov 25, 2023

Uh oh!

boegel commented Nov 25, 2023

Uh oh!

test-eessi-bot-software-layer Bot commented Nov 25, 2023

Uh oh!

boegel commented Nov 25, 2023

Uh oh!

test-eessi-bot-software-layer Bot commented Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

test-eessi-bot-software-layer Bot commented Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

test-eessi-bot-software-layer Bot commented Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Nov 25, 2023

Uh oh!

test-eessi-bot-software-layer Bot commented Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

test-eessi-bot-software-layer Bot commented Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

test-eessi-bot-software-layer Bot commented Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Nov 25, 2023

Uh oh!

test-eessi-bot-software-layer Bot commented Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Nov 25, 2023

Uh oh!

boegel commented Nov 25, 2023

Uh oh!

boegel commented Nov 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

boegel commented Nov 25, 2023 •

edited

Loading

test-eessi-bot-software-layer Bot commented Nov 25, 2023 •

edited

Loading

test-eessi-bot-software-layer Bot commented Nov 25, 2023 •

edited

Loading

test-eessi-bot-software-layer Bot commented Nov 25, 2023 •

edited

Loading

boegel commented Nov 25, 2023 •

edited

Loading

test-eessi-bot-software-layer Bot commented Nov 25, 2023 •

edited

Loading

test-eessi-bot-software-layer Bot commented Nov 25, 2023 •

edited

Loading

test-eessi-bot-software-layer Bot commented Nov 25, 2023 •

edited

Loading

test-eessi-bot-software-layer Bot commented Nov 25, 2023 •

edited

Loading