enhancements/fixes for eessi_container.sh script by trz42 · Pull Request #232 · EESSI/software-layer

trz42 · 2023-02-18T21:35:30Z

This adds changes that are in use for the NESSI bot instances. Among the changes

new arg -l | --list-repos to obtain a list of configured CernVM-FS repository identifiers
args to specify HTTP(S) proxies
http(s)_proxy env vars are used if set
if http_proxy is set, the default.local is augmented with CVMFS_HTTP_PROXY="${http_proxy}|PROXY_USING_IPv4" (the PROXY_USING_IPv4 uses the IPv4 address instead of the host name in http_proxy; it is determined by the script)
~~for easier parsing of --resume arguments it prints RESUME_FROM_DIR _DIR_ or RESUME_FROM_TGZ _PATH_TO_TGZ_~~
it stores repository information in the temporary storage to facilitate easier debugging of bot jobs

cherry-picked via commit b5bf008 ONLY picked chages to eessi_container.sh

cherry-picked via commit b5c07ee NOTE, only applied changes to eessi_container.sh

fixed bugs - determining IP address for proxy host was wrong - handling of positional parameters for executing commands with arguments was wrong

NOTE only applied changes to eessi_container.sh

cherry-picked via commit fce504f NOTE, only changed eessi_container.sh

cherry-picked via commit 23e773c NOTE only applied changes to eessi_container.sh

cherry-picked via commit bfb1b29 Note, only part for eessi_container.sh included here

boegel · 2023-02-20T10:26:25Z

  EESSI_HOST_STORAGE=$(mktemp -d --tmpdir eessi.XXXXXXXXXX)
  echo "Using ${EESSI_HOST_STORAGE} as tmp storage (add '--resume ${EESSI_HOST_STORAGE}' to resume where this session ended)."
 fi
+echo "RESUME_FROM_DIR ${EESSI_HOST_STORAGE}"


No need for this imho, we can just grep line above?

boegel · 2023-02-20T10:27:36Z

+fi
 [[ ${VERBOSE} -eq 1 ]] && echo "SINGULARITY_CACHEDIR=${SINGULARITY_CACHEDIR}"

+# pull & convert image and reset CONTAINER


clarify this comment a bit?

explain that we want to ensure that same container was used as last time?

Done that plus significantly improved it. Should also handle much better when we are resuming from a previous session. And also handles the case that CONTAINER was already pointing to an image file.

boegel · 2023-02-20T10:32:13Z


+# pull & convert image and reset CONTAINER
+CONTAINER_URL_FMT=".*://(.*)"
+if [[ ${CONTAINER} == ${CONTAINER_URL_FMT} ]]; then


should be using =~ here?

Because we use wildcards only, == should work too. However, using =~ might be safer anyway. Will change it.

boegel · 2023-02-20T10:33:56Z

  cfg_load ${EESSI_REPOS_CFG_FILE}

+  # copy repos.cfg to job directory --> makes it easier to inspect the job
+  cp ${EESSI_REPOS_CFG_FILE} ${EESSI_TMPDIR}/repos_cfg/.


cp -a ${EESSI_REPOS_CFG_FILE} ${EESSI_TMPDIR}/repos_cfg/

boegel · 2023-02-20T10:38:59Z

+    [[ ${VERBOSE} -eq 1 ]] && echo "HTTP_PROXY_IPV4='${HTTP_PROXY_IPV4}'"
+    echo "CVMFS_HTTP_PROXY=\"${http_proxy}|http://${HTTP_PROXY_IPV4}:${PROXY_PORT}\"" \
+       >> ${EESSI_TMPDIR}/repos_cfg/default.local
+    cat ${EESSI_TMPDIR}/repos_cfg/default.local


if VERBOSE, also add echo "contents of .../default.local

Added and only printing when VERBOSE is set.

boegel · 2023-02-20T10:40:47Z

  fi
  tar cf ${TGZ} -C ${EESSI_TMPDIR} .
  echo "Saved contents of '${EESSI_TMPDIR}' to '${TGZ}' (to resume, add '--resume ${TGZ}')"
+  echo "RESUME_FROM_TGZ ${TGZ}"


no need for this, we can grep for --resume in line above

boegel · 2023-02-20T10:43:09Z

+                echo "[EESSI/2021.12]" > cfg/repos.cfg
+                echo "repo_version = 2021.12" >> cfg/repos.cfg
+                echo "[EESSI/2023.02]" >> cfg/repos.cfg
+                echo "repo_version = 2023.02" >> cfg/repos.cfg


maybe use 20XY.AB instead?

trz42 · 2023-02-21T20:28:39Z

Implemented first round of requested changes in commit 47cc1c6

boegel · 2023-02-24T12:23:49Z

+    # pull container to ${EESSI_TMPDIR} if it is not there yet (i.e. when
+    # resuming from a previous session)
+    if [[ ! -x ${EESSI_TMPDIR}/${CONTAINER_IMG} ]]; then
+        singularity pull ${EESSI_TMPDIR}/${CONTAINER_IMG} ${CONTAINER}


There will be situations in which you do want to re-pull the container image rather than using the existing one, for example if a fix was deployed for the container image.

So we should have a separate option for this, to enforce re-pulling the image? (future work, not in this PR)

Make sure that quiet option is also used here unless --verbose was used?

boegel · 2023-02-24T12:28:41Z

+    # pull container to ${EESSI_TMPDIR} if it is not there yet (i.e. when
+    # resuming from a previous session)
+    if [[ ! -x ${EESSI_TMPDIR}/${CONTAINER_IMG} ]]; then
+        singularity pull ${EESSI_TMPDIR}/${CONTAINER_IMG} ${CONTAINER}


Make sure that quiet option is also used here unless --verbose was used?

…ontainer image

…ync-1 Final steps in Syncing with EESSI

truib added 19 commits February 18, 2023 20:14

added a function to obtain full path to a tool

b1a3635

added function to determine IPv4 address

27bc203

various updates for bot/build.sh and eessi_container.sh

bfe6d36

cherry-picked via commit b5bf008 ONLY picked chages to eessi_container.sh

provide CVMFS_HTTP_PROXY if necessary

8372341

cherry-picked via commit b5c07ee NOTE, only applied changes to eessi_container.sh

add functions to derive hostname and port number from proxy URL

3f1aed9

add function to determine config sections

b7abfc6

add parameter to list repositories, fix bugs

9b7b8a4

fixed bugs - determining IP address for proxy host was wrong - handling of positional parameters for executing commands with arguments was wrong

replace FILE with DIR in EESSI_REPOS_CFG_FILE_OVERRIDE

4a414f5

NOTE only applied changes to eessi_container.sh

fix whitespace issues

37abf4f

include changes made to test PR for building with the bot

d468a1a

cherry-picked via commit fce504f NOTE, only changed eessi_container.sh

fix issues in eessi_container.sh

af658ee

fix RUN_QUIET setting

eb66e1c

improvements to resuming job environment

1cb4674

cherry-picked via commit 23e773c NOTE only applied changes to eessi_container.sh

address requested change

0c465da

cleaning up leftovers after adding updates from NESSI

26ab98c

moving cfg_files.sh to scripts dir

a44088e

improved handling of container cache + explicit pull of image

bbbad0d

cherry-picked via commit bfb1b29 Note, only part for eessi_container.sh included here

use VERBOSE instead of INFO

15ae58d

fix test for --list-repos; improve output for --list-repos

764e713

boegel requested changes Feb 20, 2023

View reviewed changes

implemented requested changes

47cc1c6

trz42 mentioned this pull request Feb 22, 2023

bot/build.sh using eessi_container.sh #233

Merged

boegel reviewed Feb 24, 2023

View reviewed changes

boegel requested changes Feb 24, 2023

View reviewed changes

boegel added 2 commits February 24, 2023 13:33

add verbose messages on pulling/copying/reusing of container image

6974e4b

don't require --verbose for info message on pulling/copying/reusing c…

54bdea5

…ontainer image

boegel force-pushed the update_for_eessi_container_sh branch from 9a615bf to c001fea Compare February 24, 2023 12:58

add check to make sure that container image exists in tmpdir

a4cea9a

boegel force-pushed the update_for_eessi_container_sh branch from c001fea to a4cea9a Compare February 24, 2023 13:19

fix determining filename from container URL

c975832

boegel changed the title ~~Update for script eessi_container.sh~~ enhancements/fixes for eessi_container.sh script Feb 24, 2023

boegel approved these changes Feb 24, 2023

View reviewed changes

boegel merged commit 7ea7058 into EESSI:main Feb 24, 2023

trz42 deleted the update_for_eessi_container_sh branch March 16, 2023 10:44

TopRichard added a commit to TopRichard/software-layer that referenced this pull request Jan 19, 2024

Merge pull request EESSI#232 from TopRichard/nessi.no-2023.06-EESSI-S…

602e025

…ync-1 Final steps in Syncing with EESSI

Conversation

trz42 commented Feb 18, 2023 • edited by boegel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trz42 commented Feb 21, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

trz42 commented Feb 18, 2023 •

edited by boegel

Loading