New `diagnose` script

[mboisson]

https://github.com/ComputeCanada/software-stack-custom/blob/main/bin/computecanada/diagnose.py, a diagnose script with the following interface:

diagnose <subcommand> <parameters>

Subcommands to implement:

• job_script <job_id> => displays the job script
• inspect <path> => display the metadata of a path, for a directory, file ACLs, ls -d <path>, for a file: file ACLs, output of file <path>, `ls -d
• show <path> => display the content of a path, for a directory, display the listing, for a file, show the content
• env => display the environment in a login shell
  ...

== Restrictions ==
It must be read-only.
It must run as the user (i.e. assume no privilege)

[ofisette]

The easiest implementation of job_script <job_id> is scontrol write batch_script <job_id> -. This should be run as the user and requires no privileges. It writes to stdout (due to the trailing -), so it is effectively read-only since it does not write any files or affect the job in any way.

[plstonge]

What would be the difference between diagnose env and just env? Would it "prettify" the content of PATH (that is one line per path) and other long variables? Would it sort the SLURM variables?

One thing users would like to diagnose, is their priority / long waiting time in the queue:

• Because the squeue commands can sometimes be quite long (with formatting options, for example squeue -t pending -o "%.10i %.10j %.24P %.4a %.6C %.6D %.8m %.V %.9Q" --sort="-p" | less), having a shorter wrapper command would be helpful. That said, I do not know if users can see other users' jobs.
• Can the user specify a job ID and get some statistics about the other jobs of higher priority (within similar partitions), as well as some explanation of the "REASON"?
• Could the tool detect ill-formed jobs?
• Being able to get some diagnosis from the sshare command could also help. For example, by providing a job ID, the tool would extract the account and get the corresponding summarized sshare information (who recently used the account more than you with the percentages of the recent usage, the group LevelFS, etc.).

[mboisson]

The job_script part is implemented, I defined the structure of the code so that other features can be added more easily with PRs.

https://github.com/ComputeCanada/software-stack-custom/blob/main/bin/computecanada/diagnose.py

[mboisson]

What would be the difference between diagnose env and just env? Would it "prettify" the content of PATH (that is one line per path) and other long variables? Would it sort the SLURM variables?

One thing is that the diagnose script can be run through ccsudo, without becoming the user, therefore allowing to check a user's env without becoming them.

[ccoulombe]

I would rename the script before it is too late, something like checkenv, reportenv, would be less misleading in my opinion. Or simply check or report since it would run under ccsudo. Unless mistaken, the script will not do diagnosis but the staff would.

I'll start adding a lot tomorrow, as I've already have bits of this locally

[mboisson]

checkenv or reportenv seems far too restrictive to me. The environment is only part of the answer.

[mboisson]

The script could actually evolve to diagnose errors, if we want it to.

For example, when displaying the job script, it could highlights things like:

• failing to load modules in the job
• loading modules without version numbers
• sleep command
• conda
• failing to use a virtualenv while using python
• requesting some options that should not be used together (say, --mem=5GB and --tasks-per-node)

[ccoulombe]

Lets make this work!

[mboisson]

inspect command now works with 24303ce

Examples:

./diagnose.py inspect /usr/lib64/libX11.so.6.4.0
=======
/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/bin/ls -ld /usr/lib64/libX11.so.6.4.0
-rwxr-xr-x. 1 root root 1354984 Mar 13  2025 /usr/lib64/libX11.so.6.4.0

=======
=======
/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/bin/getfacl /usr/lib64/libX11.so.6.4.0
# file: usr/lib64/libX11.so.6.4.0
# owner: root
# group: root
user::rwx
group::r-x
other::r-x


=======
=======
/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/bin/file /usr/lib64/libX11.so.6.4.0
/usr/lib64/libX11.so.6.4.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ce36cdb38171e1e38049d284103d405b15763184, stripped

=======
=======
/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/bin/patchelf --print-rpath /usr/lib64/libX11.so.6.4.0


=======
=======
/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/bin/ldd /usr/lib64/libX11.so.6.4.0
	linux-vdso.so.1 (0x00007fff367e9000)
	libxcb.so.1 => /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libxcb.so.1 (0x00007efd45a68000)
	libc.so.6 => /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6 (0x00007efd45893000)
	libXau.so.6 => /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libXau.so.6 (0x00007efd4588e000)
	libXdmcp.so.6 => /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libXdmcp.so.6 (0x00007efd45886000)
	/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/lib64/ld-linux-x86-64.so.2 (0x00007efd45c08000)

=======

=======
/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/bin/ls -ld diskusage_explorer
-rwxrwxr-x. 1 mboisson mboisson 1819 Feb 13  2023 diskusage_explorer

=======
=======
/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/bin/getfacl diskusage_explorer
# file: diskusage_explorer
# owner: mboisson
# group: mboisson
user::rwx
group::rwx
other::r-x


=======
=======
/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/bin/file diskusage_explorer
diskusage_explorer: Bourne-Again shell script, Unicode text, UTF-8 text executable

=======

[mboisson]

show option is done with commit 4055da8

[mboisson]

env is done with 72248a3

[mboisson]

Any other ideas to implement ?