Skip to content

Add pre-install hook that raises an error in case keeppreviousinstall (used for rebuilds) is set but the installation directory still contains files#881

Closed
bedroge wants to merge 2 commits intoEESSI:2023.06-software.eessi.iofrom
bedroge:hook_check_for_nonempty_installdir
Closed

Add pre-install hook that raises an error in case keeppreviousinstall (used for rebuilds) is set but the installation directory still contains files#881
bedroge wants to merge 2 commits intoEESSI:2023.06-software.eessi.iofrom
bedroge:hook_check_for_nonempty_installdir

Conversation

@bedroge
Copy link
Copy Markdown
Collaborator

@bedroge bedroge commented Jan 23, 2025

This is an enhancement for #871 and implements the comment from @boegel in #871 (comment).

I've tested it locally with EESSI-extend for the following scenarios:

  • no keeppreviousinstall -> this always works, as expected
  • with --try-amend=keeppreviousinstall=True:
    • empty install dir -> works
    • only some empty subdirs in the install dir -> works
    • file in the root of install dir -> error
    • file in a subdir -> error

You can test it by loading EESSI-extend, overriding EASYBUILD_HOOKS and setting it to the hooks file of this PR, and doing a small build with or without --try-amend=keeppreviousinstall=True.

@bedroge bedroge added the enhancement New feature or request label Jan 23, 2025
@eessi-bot
Copy link
Copy Markdown

eessi-bot Bot commented Jan 23, 2025

Instance eessi-bot-mc-aws is configured to build for:

  • architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/sapphire_rapids, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
  • repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

@eessi-bot
Copy link
Copy Markdown

eessi-bot Bot commented Jan 23, 2025

Instance eessi-bot-mc-azure is configured to build for:

  • architectures: x86_64/amd/zen4
  • repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

@bedroge bedroge mentioned this pull request Jan 23, 2025
Merged
@boegel boegel added the 2023.06-software.eessi.io 2023.06 version of software.eessi.io label Jan 24, 2025
@boegel
Copy link
Copy Markdown
Contributor

boegel commented Jan 24, 2025

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

@eessi-bot
Copy link
Copy Markdown

eessi-bot Bot commented Jan 24, 2025 •
edited
Loading

Updates by the bot instance eessi-bot-mc-aws (click for details)

@eessi-bot
Copy link
Copy Markdown

eessi-bot Bot commented Jan 24, 2025 •
edited
Loading

Updates by the bot instance eessi-bot-mc-azure (click for details)

  • received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from boegel

    • expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2

  • handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:

    • no jobs were submitted

@eessi-bot
Copy link
Copy Markdown

eessi-bot Bot commented Jan 24, 2025 •
edited
Loading

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2025.01/pr_881/42405

date job status comment
Jan 24 11:24:19 UTC 2025 submitted job id 42405 awaits release by job manager
Jan 24 11:24:53 UTC 2025 released job awaits launch by Slurm scheduler
Jan 24 11:25:55 UTC 2025 running job 42405 is running
Jan 24 11:33:02 UTC 2025 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-42405.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-1737717953.tar.gzsize: 0 MiB (14895 bytes)
entries: 1
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
other under 2023.06/software/linux/x86_64/amd/zen2
2023.06/init/easybuild/eb_hooks.py
Jan 24 11:33:02 UTC 2025 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default
P: perf: 441.115 timesteps/s (r:0, l:None, u:None)
[ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default
P: perf: 427.9 timesteps/s (r:0, l:None, u:None)
[ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 1.92 us (r:0, l:None, u:None)
[ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 1.77 us (r:0, l:None, u:None)
[ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 3.84 us (r:0, l:None, u:None)
[ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 4.09 us (r:0, l:None, u:None)
[ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 0.57 us (r:0, l:None, u:None)
[ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 0.58 us (r:0, l:None, u:None)
[ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86-64-amd-zen2-node+default
P: bandwidth: 7444.26 MB/s (r:0, l:None, u:None)
[ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86-64-amd-zen2-node+default
P: bandwidth: 7388.1 MB/s (r:0, l:None, u:None)
[ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-42405.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

boegel
boegel requested changes Jan 24, 2025
View reviewed changes
Comment thread eb_hooks.py

def pre_install_hook(self, *args, **kwargs):
"""Main pre install hook: trigger custom functions based on software name."""
if 'keeppreviousinstall' in self.cfg and self.cfg['keeppreviousinstall']:
Copy link
Copy Markdown
Contributor

@boegel boegel Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bedroge I think we deserve some context here, where/how we're using keeppreviousinstall, and why we only want to use this only when empty directories are available.
Definitely point to #556 + #871

Copy link
Copy Markdown
Collaborator Author

@bedroge bedroge Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: cc79fd7

@laraPPr
Copy link
Copy Markdown
Collaborator

laraPPr commented Jun 27, 2025

@bedroge can you move these changes to https://github.com/EESSI/software-layer-scripts

@laraPPr laraPPr closed this Jun 27, 2025
@bedroge bedroge deleted the hook_check_for_nonempty_installdir branch July 18, 2025 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boegel boegel boegel requested changes

Assignees

No one assigned

Labels

2023.06-software.eessi.io 2023.06 version of software.eessi.io enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@bedroge @boegel @laraPPr