user-friendlyer job runs

[martinkilbinger]

Summary

More user-friendly job runs.

Remove necessity of symbolic links in job script, by

• re-organisation of config files and runs
• using functions in file_handler that were quick-and-dirty copied from run_log. This allows to get paths to previous runs in arbitrary config entries. This is just a quick-and-dirty work around. Let's discuss this. I'm sure we can find a better solution!

Reviewer Checklist

• The PR targets the develop branch
• The PR is assigned to the developer
• The PR has appropriate labels
• The PR is included in appropriate projects and/or milestones
• The PR includes a clear description of the proposed changes
• If the PR addresses an open issue the description includes "closes #"
• The code and documentation style match the current standards
• Documentation has been added/updated consistently with the code
• All CI tests are passing
• API docs have been built and checked at least once (if relevant)
• All changed files have been checked and comments provided to the developer
• All of the reviewer's comments have been satisfactorily addressed by the developer

[sfarrens]

@martinkilbinger following today's discussion I propose that you do the following:

• In run_log.py

  • abstract the RunLog class method (get_list, get_all, get_last) as stand-alone functions
  • make run_log_file an input parameter
  • create a new function get_last_dir that calls get_last

• In file_handler.py

  • import the newly abstracted functions from run_log.py
  • replace all references to self._run_log with the abstracted functions
  • use self._run_log_file to specify the run_log_file input parameter
  • in _create_module_run_dirs update self.module_run_dirs as follows

self.module_run_dirs = {
    'run': self.run_dir,
    'log': self._log_dir,
    'tmp': self._tmp_dir,
    'output': self.output_dir,
    'run_log': self._run_log_file
}

• In the module runners
  • import get_last_dir from run_log.py
  • use run_dirs['run_log'] to specify the run_log_file input parameter

[sfarrens]

@martinkilbinger I think you are missing a few changes to file_handler.py to use the newly abstracted functions in run_log.py.

[sfarrens]

@martinkilbinger I think this is nearly done now. I can run some tests once you fix the last couple of comments and the build the API docs to check that everything looks good.

[sfarrens]

That's too bad, would be useful to know what type of list is returned.
Ah well, I changed it, having the link option is also nice.

OK @martinkilbinger last thing! I built the docs locally to check everything and it looks like the intersphinx mapping (the tool that links data types to the appropriate API documentation) breaks when you put list of str etc.

See, for example, that str is a link while list of str is not. To fix this I suggest you update the new functions in run_log.py and for the input and return data types just put list.