From @TaiSakuma : add a Job option for a post-execution script, before files are transferred to HDFS (à la setup_script).
An example workflow would be: setup (conda activate) => run python script => postrun (zip up output) => transfer to HDFS.
Thinking of calling it postrun_script. Other suggestions: teardown_script, although this is a misnomer as (a) the user can use it for anything, (b) it isn't necessary to remove anything from the worker node - the script does this automatically
From @TaiSakuma : add a
Joboption for a post-execution script, before files are transferred to HDFS (à la setup_script).An example workflow would be: setup (conda activate) => run python script => postrun (zip up output) => transfer to HDFS.
Thinking of calling it
postrun_script. Other suggestions:teardown_script, although this is a misnomer as (a) the user can use it for anything, (b) it isn't necessary to remove anything from the worker node - the script does this automatically