Multiple Module Runs

README.rst

@@ -9,9 +9,9 @@ ShapePipe
 .. |python38| image:: https://img.shields.io/badge/python-3.8-green.svg
   :target: https://www.python.org/
 
-:Version: 0.0.3
+:Version: 0.0.4
 
-:Date: 19/05/2020
+:Date: 15/07/2021
 
 ShapePipe is a galaxy shape measurement pipeline developed within the
 CosmoStat lab at CEA Paris-Saclay.
@@ -161,14 +161,14 @@ The pipeline can be run with SMP as follows:
 
 .. code-block:: bash
 
-  $ ./shapepipe_run
+  $ shapepipe_run
 
 A list of command line arguments can be displayed using the ``--help``
 option:
 
 .. code-block:: bash
 
-  $ ./shapepipe_run --help
+  $ shapepipe_run --help
 
 Running the Pipeline with MPI
 -----------------------------
@@ -177,7 +177,7 @@ The pipeline can be run with MPI as follows:
 
 .. code-block:: bash
 
-  $ mpiexec -n <number_of_cores> ./shapepipe_run
+  $ mpiexec -n <number_of_cores> shapepipe_run
 
 where ``<number_of_cores>`` is the number of cores to allocate to the run.
 
@@ -213,13 +213,18 @@ The configuration parameters for the pipeline are:
    ``shapepipe_runs``.
 3. ``INPUT_DIR`` : (``str`` or ``list``) A valid directory containing input
    files for the first module or a comma separated list of directories. This
-   parameter also recognizes the following special strings:
+   parameter also recognises the following special strings:
 
    a. ``last:MODULE`` : This will point to the output directory of the last run
       of the specified module.
-   b. ``PATTERN:MODULE`` : This will point to the output directory of a
+   b. ``all:MODULE`` : This will point to all the output directories in which
+      the specified module was run.
+   c. ``PATTERN:MODULE`` : This will point to the output directory of a
       specified module from a run matching the specified pattern.
 
+   In all cases the module name can be succeded by the run number (*e.g.*
+   ``MODULE_run_2``)
+
 4. ``OUTPUT_DIR`` : (``str``) A valid directory to write the pipeline output
    files.
 5. ``FILE_PATTERN`` : (``str`` or ``list``) A list of string patterns to
@@ -256,6 +261,17 @@ additional values to list properties as follows:
   [MODULE_NAME]
   ADD_PARAMETER = PARAMETER VALUE
 
+If a given module is run more than once, run specific parameter values can be
+specified as follows:
+
+.. code-block:: bash
+
+  [MODULE_NAME_RUN_X]
+  PARAMETER = PARAMETER VALUE
+
+Where ``X`` is an integer greater than or equal to ``1``.
+
+
 Development
 ===========
 
@@ -271,7 +287,7 @@ with the ``module_runner`` wrapper that outputs the module ``stdout`` and
 
 .. code-block:: python
 
-  @module_runner()
+  @module_runner(version=0.1)
   def example_module(*args)
 
     # DO SOMETHING
@@ -282,7 +298,9 @@ The module runner decorator takes the following keyword arguments:
 
 1. ``input_module`` :  (``str`` or ``list``) The name of a preceding module(s)
    whose output provide(s) the input to this module. Default value is ``None``.
-2. ``version`` : (``str``) The module version. Default value is ``'0.0'``.
+2. ``version`` : (``str``) The module version. Default value is ``'0.0'``. The
+   module version should always be explicitly declared and be greater than the
+   default value.
 3. ``file_pattern`` : (``str`` or ``list``) The input file pattern(s) to look
    for. Default value is ``''``.
 4. ``file_ext`` : (``str`` or ``list``) The input file extensions(s) to look
@@ -297,7 +315,8 @@ The module runner decorator takes the following keyword arguments:
 The arguments passed to the module runner are the following:
 
 1. ``input_file_list`` : The list of input files.
-2. ``output_dir`` : The directory for the module output files.
+2. ``run_dirs`` : The dictionary containing module run paths (*e.g.* the output
+   path for a given module run is ``run_dirs['output']``).
 3. ``file_number_string`` : The number pattern corresponding to the current
    process.
 4. ``config`` : The config parser instance, which provides access to the
@@ -308,7 +327,9 @@ The arguments passed to the module runner are the following:
 
      parameter_value = config.get('MODULE_NAME', 'PARAMETER')
 
-5. ``w_log`` : The worker log instance, which can be used to record additional
+5  ``module_config_sec`` : A string specifying the configuration file section
+   to be read.
+6. ``w_log`` : The worker log instance, which can be used to record additional
    messages in the module output logs using the following structure:
 
    .. code-block:: python
@@ -322,11 +343,10 @@ The following example module runners are provided in ``shapepipe.modules``.
 
 **Python Example**
 
-In this example a Python script using a ``Dummy()`` class is implemented. This
-module does not read inputs from any preceding module, but looks for files
+In this example a Python script using a ``PythonExample()`` class is implemented.
+This module does not read inputs from any preceding module, but looks for files
 in the ``INPUT_DIR`` that match the file patterns ``'numbers'`` and
-``'letters'`` with file extension ``'.txt'``. This module depends on
-``numpy``.
+``'letters'`` with file extension ``'.txt'``. This module depends on ``numpy``.
 
 As this module does not implement any system executable, it is not
 necessary to return a ``stderr``. Instead any output content that should be
@@ -335,20 +355,31 @@ return ``None, None``.
 
 .. code-block:: python
 
-  @module_runner(version='1.0', file_pattern=['numbers', 'letters'],
-                 file_ext='.txt', depends='numpy')
-  def python_example(input_file_list, output_dir, file_number_string,
-                     config, w_log):
-
-      output_file_name = ('{}/pyex_output{}.cat'.format(output_dir,
-                          file_number_string))
-      message = config.get('PYTHON_EXAMPLE', 'MESSAGE')
-
-      inst = Dummy()
-      inst.read_files(*input_file_list)
-      inst.write_file(output_file_name, message)
-
-      return inst.content, None
+  @module_runner(
+    version='1.1',
+    file_pattern=['numbers', 'letters'],
+    file_ext='.txt',
+    depends='numpy',
+  )
+  def python_example_runner(
+    input_file_list,
+    run_dirs,
+    file_number_string,
+    config,
+    module_config_sec,
+    w_log,
+  ):
+
+    output_file_name = (
+        f'{run_dirs["output"]}/pyex_output{file_number_string}.cat'
+    )
+    message = config.get(module_config_sec, 'MESSAGE')
+
+    inst = pe.PythonExample(0)
+    inst.read_files(*input_file_list)
+    inst.write_file(output_file_name, message)
+
+    return inst.content, None
 
 **Executable Example**
 
@@ -358,17 +389,30 @@ the file pattern ``'process'`` with file extension ``'.cat'``.
 
 .. code-block:: python
 
-  @module_runner(input_module='python_example', version='1.0',
-                 file_pattern='pyex_output', file_ext='.cat', executes='head')
-  def execute_example(input_file_list, output_dir, file_number_string, *args):
-
-      command_line = 'head {}'.format(input_file_list[0])
-      output_file_name = '{}/head_output{}.txt'.format(output_dir,
-                                                       file_number_string)
+  @module_runner(
+    input_module='python_example',
+    version='1.0',
+    file_pattern='pyex_output',
+    file_ext='.cat',
+    executes='head',
+  )
+  def execute_example_runner(
+    input_file_list,
+    run_dirs,
+    file_number_string,
+    config,
+    module_config_sec,
+    w_log,
+  ):
+
+    command_line = f'head {input_file_list[0]}'
+    output_file_name = (
+        f'{run_dirs["output"]}/head_output{file_number_string}.txt'
+    )
+
+    stdout, stderr = execute(command_line)
+
+    text_file = open(output_file_name, 'w')
+    text_file.write(stdout)
 
-      stdout, stderr = execute(command_line)
-
-      text_file = open(output_file_name, 'w')
-      text_file.write(stdout)
-
-      return stdout, stderr
+    return stdout, stderr

example/config.ini

@@ -12,7 +12,7 @@
 ## ShapePipe execution options
 [EXECUTION]
 # MODULE (required) must be a valid module runner name (or a comma separated list of names)
-MODULE = python_example, serial_example, execute_example
+MODULE = python_example_runner, serial_example_runner, execute_example_runner, python_example_runner, execute_example_runner
 # MODE (optional) options are smp or mpi, default is smp
 ; MODE = mpi
 
@@ -54,8 +54,18 @@ TIMEOUT = 00:01:35
 ; PROCESS_PRINT_LIMIT = 100
 
 ## Module options
-[PYTHON_EXAMPLE]
+[PYTHON_EXAMPLE_RUNNER_RUN_1]
 MESSAGE = The obtained value is:
 
-[SERIAL_EXAMPLE]
+[SERIAL_EXAMPLE_RUNNER]
 ADD_INPUT_DIR = ./example/data/numbers, ./example/data/letters
+
+[PYTHON_EXAMPLE_RUNNER_RUN_2]
+INPUT_MODULE = python_example_runner_run_1, execute_example_runner
+FILE_PATTERN = pyex_output, head_output
+FILE_EXT = cat, txt
+
+MESSAGE = The new obtained value is:
+
+[EXECUTE_EXAMPLE_RUNNER_RUN_2]
+INPUT_MODULE = python_example_runner_run_2

shapepipe/info.py

@@ -17,31 +17,36 @@
 
 
 # Package Info
-version_info = (0, 0, 2)
+version_info = (0, 0, 4)
 __version__ = '.'.join(str(c) for c in version_info)
 __name__ = 'shapepipe'
 __author__ = 'Samuel Farrens'
 __email__ = 'samuel.farrens@cea.fr'
-__about__ = ('ShapePipe is a shape measurement pipeline developed with the'
-             'CosmoStat lab at CEA Paris-Saclay.')
+__about__ = (
+    'ShapePipe is a shape measurement pipeline developed with the'
+    + 'CosmoStat lab at CEA Paris-Saclay.'
+)
 __setups__ = ['pytest-runner']
-__installs__ = ['joblib>=0.13',
-                'modopt>=1.2',
-                'numpy>=1.14']
-__tests__ = ['pytest',
-             'pytest-cov',
-             'pytest-pycodestyle']
+__installs__ = [
+    'joblib>=0.13',
+    'modopt>=1.2',
+    'numpy>=1.14'
+]
+__tests__ = [
+    'pytest',
+    'pytest-cov',
+    'pytest-pycodestyle'
+]
 
 
 def shapepipe_logo(colour=False):
-    """ ShapePipe Logo
+    """ShapePipe Logo
 
     Returns
     -------
     str logo string
 
     """
-
     shape = r'''
  _______  __   __  _______  _______  _______  _______  ___   _______  _______
 |       ||  | |  ||   _   ||       ||       ||       ||   | |       ||       |
@@ -79,14 +84,13 @@ def shapepipe_logo(colour=False):
 
 
 def line():
-    """ Line
+    """Line
 
     Returns
     -------
     str a horizontal line
 
     """
-
     line = r'''
 -------------------------------------------------------------------------------
     '''

shapepipe/modules/__init__.py

@@ -8,37 +8,46 @@
 
 """
 
-__all__ = ['module_decorator',
-           'module_runners',
-           'mask_package',
-           'PSFExInterpolation_package',
-           'SETools_package'
-           ]
-__module_list__ = ['execute_example',
-                   'erase_output_runner',
-                   'find_exposures_runner',
-                   'galsim_shapes_runner',
-                   'galsim_shapes_v2_runner',
-                   'get_images_runner',
-                   'make_catalog_runner',
-                   'mask_runner',
-                   'mask_runner_exp',
-                   'match_external_runner',
-                   'merge_sep_cats_runner',
-                   'merge_headers_runner',
-                   'merge_star_cat_runner',
-                   'ngmix_runner',
-                   'paste_cat_runner',
-                   'psfex_runner',
-                   'psfex_interp_runner',
-                   'python_example',
-                   'serial_example',
-                   'setools_runner',
-                   'sextractor_runner',
-                   'sextractor_runner_exp',
-                   'split_exp_runner',
-                   'spread_model_runner',
-                   'swarp_runner',
-                   'vignetmaker_runner',
-                   'vignetmaker_runner2',
-                   'uncompress_fits_image_runner']
+__all__ = [
+    'find_exposures_package',
+    'mask_package',
+    'MCCD_package',
+    'module_decorator',
+    'module_runners',
+    'mask_package',
+    'PasteCat_package'
+    'PSFExInterpolation_package',
+    'python_example_package',
+    'SETools_package',
+    'vignetmaker_package',
+]
+__module_list__ = [
+    'execute_example',
+    'erase_output_runner',
+    'find_exposures_runner',
+    'galsim_shapes_runner',
+    'galsim_shapes_v2_runner',
+    'get_images_runner',
+    'get_images_runner2',
+    'make_catalog_runner',
+    'mask_runner',
+    'mask_runner_exp',
+    'match_external_runner',
+    'merge_sep_cats_runner',
+    'merge_headers_runner',
+    'merge_star_cat_runner',
+    'ngmix_runner',
+    'paste_cat_runner',
+    'psfex_runner',
+    'psfex_interp_runner',
+    'python_example',
+    'serial_example',
+    'setools_runner',
+    'sextractor_runner',
+    'sextractor_runner_exp',
+    'split_exp_runner',
+    'spread_model_runner',
+    'swarp_runner',
+    'vignetmaker_runner',
+    'uncompress_fits_image_runner',
+]