Make it easier (more obious how) to use github features when using EasyBuild on a shared system with EESSI

[adammccartney]

Trying to --upload-test-report when building a piece of software on a shared system using EESSI-extend is really difficult.

Tried to follow the docs, which advise to use --install-github-token, which requires pip install keyring and (python-pass + pass) to set up a backend. Pass uses gpg for decryption and signing, so it's not all that obvious that we want to leave a key with those sorts of abilities lying around on a shared system. Whatever, there is also the unsafe option of installing keyrings.alt which lets us use plaintext or light encryption on a file. Good, that's fine we can delete the file after we upload the bug report.

The questions is where to install those pacakges, assuming we're on a system using easybuild from EESSI-extend.
Tried to install them into a dedicated virtualenv, and it is activated in before the call below, but fails with the following output:

> eb --github-user adammccartney --install-github-token
== Temporary log file in case of crash /tmp/eb-loqmolm5/easybuild-pj5wz1gg.log
Token:
Validating token...
Token seems to be valid, installing it.
EasyBuild crashed! Please consider reporting a bug, this should not happen...

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen4/software/EasyBuild/5.1.0/lib/python3.11/site-packages/easybuild/main.py", line 829, in <module>
    main_with_hooks()
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen4/software/EasyBuild/5.1.0/lib/python3.11/site-packages/easybuild/main.py", line 815, in main_with_hooks
    main(args=args, prepared_cfg_data=(init_session_state, eb_go, cfg_settings))
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen4/software/EasyBuild/5.1.0/lib/python3.11/site-packages/easybuild/main.py", line 691, in main
    install_github_token(options.github_user, silent=build_option('silent'))
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen4/software/EasyBuild/5.1.0/lib/python3.11/site-packages/easybuild/tools/github.py", line 2614, in install_github_token
    keyring.set_password(KEYRING_GITHUB_TOKEN, github_user, token)
  File "/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib/python3.11/site-packages/keyring/core.py", line 60, in set_password
    get_keyring().set_password(service_name, username, password)
    ^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib/python3.11/site-packages/keyring/core.py", line 32, in get_keyring
    init_backend()
  File "/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib/python3.11/site-packages/keyring/core.py", line 83, in init_backend
    set_keyring(_detect_backend(limit))
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib/python3.11/site-packages/keyring/core.py", line 97, in _detect_backend
    load_env()
  File "/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib/python3.11/site-packages/keyring/core.py", line 143, in load_env
    return load_keyring(os.environ['PYTHON_KEYRING_BACKEND'])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib/python3.11/site-packages/keyring/core.py", line 134, in load_keyring
    class_ = _load_keyring_class(keyring_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib/python3.11/site-packages/keyring/core.py", line 124, in _load_keyring_class
    __import__(module_name)
ModuleNotFoundError: No module named 'keyrings'

[ocaisa]

The tricky part here is to understand that EasyBuild will always use the python you installed it with, unless you tell it to do otherwise:

# Initialise EESSI
ocaisa@~$ source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash
EESSI/2023.06 loaded successfully

# Load EESSI-extend
ocaisa@~$ module load EESSI-extend
-- Using /tmp/$USER as a temporary working directory for installations, you can override this by setting the environment variable WORKING_DIR and reloading the module (e.g., /dev/shm
is a common option)
Configuring for use of EESSI_USER_INSTALL under /home/ocaisa/eessi
-- To create installations for EESSI, you _must_ have write permissions to /home/ocaisa/eessi/versions/2023.06/software/linux/x86_64/intel/icelake
-- You may wish to configure a sources directory for EasyBuild (for example, via setting the environment variable EASYBUILD_SOURCEPATH) to allow you to reuse existing sources for
packages.

# Create a venv with the current python (from EESSI)
ocaisa@~$ python -m venv temp_venv
ocaisa@~$ source temp_venv/bin/activate
(temp_venv) ocaisa@~$ pip list
Package               Version
--------------------- -------
easybuild-easyblocks  5.1.0
easybuild-easyconfigs 5.1.0
easybuild-framework   5.1.0
pip                   23.1.2
setuptools            67.8.0

# Install required packages
(temp_venv) ocaisa@~$ pip install keyring keyrings.alt gitpython rich
...
Installing collected packages: zipp, smmap, pygments, pycparser, more-itertools, mdurl, jeepney, backports.tarfile, markdown-it-py, jaraco.functools, jaraco.context, jaraco.classes, importlib_metadata, gitdb, cffi, rich, keyrings.alt, gitpython, cryptography, SecretStorage, keyring
Successfully installed SecretStorage-3.3.3 backports.tarfile-1.2.0 cffi-1.17.1 cryptography-45.0.3 gitdb-4.0.12 gitpython-3.1.44 importlib_metadata-8.7.0 jaraco.classes-3.4.0 jaraco.context-6.0.1 jaraco.functools-4.1.0 jeepney-0.9.0 keyring-25.6.0 keyrings.alt-5.0.2 markdown-it-py-3.0.0 mdurl-0.1.2 more-itertools-10.7.0 pycparser-2.22 pygments-2.19.1 rich-14.0.0 smmap-5.0.2 zipp-3.22.0

# Check status of GitHub integration
(temp_venv) ocaisa@~$ eb --check-github --github-user=ocaisa
== Temporary log file in case of crash /tmp/eb-ggzrhl_k/easybuild-lvl8x5is.log

Checking status of GitHub integration...

Making sure we're online...OK

* GitHub user...ocaisa => OK
* GitHub token...(no token found) => FAIL
* git command...OK ("git version 2.41.0; ")
* GitPython module...OK (GitPython version 3.1.31)
...

# Didn't work as EasyBuild is not using my venv python, force this
(temp_venv) ocaisa@~$ EB_PYTHON=$(which python) eb --check-github --github-user=ocaisa
== Temporary log file in case of crash /tmp/eb-k53o15gg/easybuild-ic1s4yfc.log

Checking status of GitHub integration...

Making sure we're online...OK

* GitHub user...ocaisa => OK
* GitHub token...ghp..DwV (len: 40) => OK (validated)
* git command...OK ("git version 2.41.0; ")
* GitPython module...OK (GitPython version 3.1.44)
...

Once your venv is in place you can update the activation script for the venv to load EESSI-extend and set EB_PYTHON (and maybe EASYBUILD_GITHUB_USER)

[ocaisa]

The custom stuff for temp_venv/bin/activate could look like:

# Custom stuff
source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash
module load EESSI
module load EESSI-extend
export EB_PYTHON=/home/ocaisa/temp_venv/bin/python
export EASYBUILD_GITHUB_USER=ocaisa

and then things look like:

ocaisa@~$ source temp_venv/bin/activate
EESSI/2023.06 loaded successfully
-- Using /tmp/$USER as a temporary working directory for installations, you can override this by setting the environment variable WORKING_DIR and reloading
the module (e.g., /dev/shm is a common option)
Configuring for use of EESSI_USER_INSTALL under /home/ocaisa/eessi
-- To create installations for EESSI, you _must_ have write permissions to /home/ocaisa/eessi/versions/2023.06/software/linux/x86_64/intel/icelake
-- You may wish to configure a sources directory for EasyBuild (for example, via setting the environment variable EASYBUILD_SOURCEPATH) to allow you to reuse
existing sources for packages.

(temp_venv) ocaisa@~$ eb --check-github
== Temporary log file in case of crash /tmp/eb-3pvou5tu/easybuild-1zmxo22t.log

Checking status of GitHub integration...

Making sure we're online...OK

* GitHub user...ocaisa => OK
* GitHub token...ghp..DwV (len: 40) => OK (validated)
* git command...OK ("git version 2.41.0; ")
* GitPython module...OK (GitPython version 3.1.44)
...

[adammccartney]

Ah cool, thanks for the detailed answer!
I think my issue was not using the python from eessi's compat layer. Instead loaded the default python module, created the venv and then loaded EESSI-extend.

[adammccartney]

Actually, I wanted to ask about the installed modules in the environment. This is really cool, how are you getting a base set of modules to appear in the venv?

# Create a venv with the current python (from EESSI)
ocaisa@~$ python -m venv temp_venv
ocaisa@~$ source temp_venv/bin/activate
(temp_venv) ocaisa@~$ pip list
Package               Version
--------------------- -------
easybuild-easyblocks  5.1.0
easybuild-easyconfigs 5.1.0
easybuild-framework   5.1.0
pip                   23.1.2
setuptools            67.8.0

Like it seems a venv doesn't automatically inherit installed modules from the python that created the venv, or am I mistaken?

[ocaisa]

It is cheating unfortunately 🤕 . The EasyBuild module sets PYTHONPATH, that's why these modules are visible by default. The negative of that is there is no may to upgrade them in the environment (the modules that appear in PYTHONPATH always take precedence).

[Crivella]

There seems to be a bit of peculiar behavior though.
The files from the original EESSI will always be picked up first, but new files defined in the venv that are not in EESSI can still be picked up.

EG: I was wondering why after loading my develop venv on top of EESSI+EESSI-extend i was still able to see the toolchains that i am working on with --list-toolchains that are not in EESSI.
This is because they are defined in files that are not in the EESSI version of EasyBuild, so python go looks for them further in the venv enforced paths

[ocaisa]

I looked a bit into sagecipher which should allow using your ssh-agent with keyring. It does work but takes a bit of configuration. In particular you need

export PYTHON_KEYRING_BACKEND="sagecipher.keyring.Keyring"

as it also pulls in keyrings.alt as a dependency.

If you want a specific key to be used you need to provide export KEYRING_PROPERTY_SSH_KEY_FINGERPRINT=e8:19:fe:c5:0a:b4:57:5d:96:27:b3:e3:ec:ba:24:3c (the hex fingerprint of the key you want to use). You could add these to your venv initialisaton so it would at least work.