determine easystack files to process via PR patch file#351
determine easystack files to process via PR patch file#351trz42 merged 2 commits intoEESSI:2023.06from
Conversation
|
Instance
|
trz42
left a comment
trz42
left a comment
There was a problem hiding this comment.
Looks good. Two minor suggestions. Feel free to ignore them.
| pr_diff=$(ls [0-9]*.diff | head -1) | ||
|
|
||
| # use PR patch file to determine in which easystack files stuff was added | ||
| for easystack_file in $(cat ${pr_diff} | grep '^+++' | cut -f2 -d' ' | sed 's@^[a-z]/@@g' | grep '^eessi.*yml$'); do |
There was a problem hiding this comment.
That's ok. However, if we can assume that git is available we could also just do
for easystack_file in $(git diff --name-only | grep '^eessi.*yml$'); doThere was a problem hiding this comment.
Wouldn't git diff also list files that were removed?
With +++ I'm making sure we only get files where stuff was added (so the file will exist).
May not matter much in practice, as long as we don't remove easystack files (I think we may do that when rebuilding for eessi.io, but we could do a cleanup PR first to collapse easystack files to latest EasyBuild release, and then follow up with "build" PRs for eessi.io).
So I'll switch to this, it's a lot cleaner (and hopefully it's only temporary, since I would really prefer getting this info from the bot instead).
| echo -e "Processing easystack file ${easystack_file}...\n\n" | ||
|
|
||
| # determine version of EasyBuild module to load based on EasyBuild version included in name of easystack file | ||
| eb_version=$(echo ${easystack_file} | sed 's/.*eb-\([0-9.]*\).*/\1/g') |
There was a problem hiding this comment.
Alternative way might be
eb_version=$(echo ${easystack_file} | cut -f 4 -d '-')There was a problem hiding this comment.
That's a bit less strict, at least with eb- we have some guarantee that which follows is an EasyBuild version...
So I prefer sticking to the sed.
Eventually we probably want to specify the EasyBuild version to use in the easystack file itself (but then EasyBuild needs to allow for that, I think).
{2023.06}[foss/2023a] CUDA samples v12.1
2 participants
We're starting to hit GitHub rate limits more often because we're using
--from-prand--include-easyblocks-from-prin various easystack files now.Currently, the
EESSI-pilot-install-software.shscript that is run bybot/build.shscript simply iterates over all existing easystack files, regardless of whether or not they have been changed, and thus triggers a whole bunch of useless EasyBuild sessions that hammer the GitHub API.These changes use the
<PR>.difffiles that is left behind in the working directory that is prepared by the bot to only process the easystack files that are actually changed in a PR, which significantly limits the EasyBuild sessions being started.I think it would be better if the bot collects the info on which files have been changed/added/removed by a PR, but for now this "dirty" approach would already help a lot, and can be replaced with a cleaner approach later if the bot feeds in a list of changed/removed/added files via the
cfg/job.cfgfile it prepares for each build job.tested via boegel#28, works as intended