Tutorial documentation

docs/source/pipeline_tutorial.md

@@ -73,20 +73,20 @@ Naming and numbering of the input files can closely follow the original image na
   files with user-defined summary statistics for each CCD, for example the number of selected stars, or mean and standard deviation of their FWHM.  
   Example: `star_stat-2366993-18.txt`
 
-- Tile ID list
+- Tile ID list  
   ASCII file with a tile number on each line. Used for the `get_image_runner` module to download CFIS images (see [Download tiles](#download-tiles)).
 
-  - Single-exposure name list
-    ASCII file with a single-exposure name on each line. Produced by the `find_exposure_runner` module to identify single exposures that were used to create
-    a given tile. See [Find exposures](#find-exposures)).
+- Single-exposure name list  
+  ASCII file with a single-exposure name on each line. Produced by the `find_exposure_runner` module to identify single exposures that were used to create
+  a given tile. See [Find exposures](#find-exposures)).
 
-- Plots
+- Plots  
   The `SETools` module can also produce plots of the objects properties that were selected for a given CCD.
   The type of plot (histogram, scatter plot, ...) and quantities to plot as well as plot decorations can be specified in the
   selection criteria config file (see [Select stars](#select-stars)).
   Example: `hist_mag_stars-2104133-5.png`
 
-- Log files
+- Log files  
   The pipeline core and all called modules write ASCII log files to disk.  
   Examples: `process-2366993-6.log`, `log_sp_exp.log`.
 
@@ -125,7 +125,7 @@ job_sp -h
 ```
 for all options.
 
-This script created the subdirectory `$SP_RUN/output` to store all pipeline outputs
+This script creates the subdirectory `$SP_RUN/output` to store all pipeline outputs
 (log files, diagnostics, statistics, output images, catalogues, single-exposure headers with WCS information).
 
 Optionally, the subdir `output_star_cat` is created by the used to store the external star catalogues for masking. This is only necessary if the pipeline is run on a cluster without internet connection to access star catalogues. In that case, the star catalogues need to be retrieved outside the pipeline, for example on a login node, and copied to `output_star_cat`.
@@ -173,6 +173,9 @@ For the retrieval method the user can choose betwen
 
 Note that internet access is required for this step if the download method is `vos`.
 
+An output directory `run_sp_GitFeGie` (in `output`) is created containing the results of `get_images` for tiles (`Git`),
+`find_exposures` (`Fe`), and `get_images` for exposures (`Gie`).
+
 ## Prepare input images
 
 With
@@ -183,6 +186,9 @@ the compressed tile weight image is uncompressed via the `uncompress_fits` modul
 (one FITS file per CCD) with `split_exp`.
 Finally, the headers of all single-exposure single-CCD files are merged into a single `sqlite` file, to store the WCS information of the input exposures.
 
+Two output directories are created, `run_sp_Uz` for `uncompress_fits`, and `run_sp_exp_SpMh` for the output of the modules
+`split_exp` (`Sp`) and `merge_headers` (`Mh`).
+
 ## Mask images
 
 Run
@@ -193,14 +199,17 @@ to mask tile and single-exposure single-CCD images. Both tasks are performed by
 
 Note that internet access is required for this step, since a reference star catalogue is downloaded.
 
+The output of both masking runs are stored in the output directory `run_sp_MaMa`, with run 1 (2) of
+`mask` corresponding to tiles (exposures).
+
 **Diagnostics:** Open a single-exposure single-CCD image and the corresponding pipeline flag
 in `ds9`, and display both frames next to each other. Example
 ```bash
-ds9 output/shapepipe_run_2020-03-03_15-31-00/split_exp_runner/output/image-2113737-10.fits output/shapepipe_run_2020-03-03_17-29-34/mask_runner/output/pipeline_flag-2113737-10.fits
+ds9 image-2113737-10.fits pipeline_flag-2113737-10.fits
 ```
 Choose `zoom fit` for both frames, click `scale zscale` for the image, and `color aips0` for the flag, to display something like this:
 
-<img width="250" src="diag_mask.png">
+<img width="250" src="./images/diag_mask.png">
 
 By eye the correspondence between the different flag types and the image can be
 seen. Note that the two frames might not match perfectly, since (a) WCS
@@ -223,21 +232,25 @@ Next, the following tasks are run on the single-exposure single-CCD images:
   via a call to `psfex_interp`. For MCCD, the modules `merge_starcat`, `mccd_plots`, and
   `mccd_interp` are called.
 
+The output directory for both the `mccd` and `psfex` options is `run_sp_tile_Sx_exp_SxSePsf`.
+This stores the output of SExtractor on the tiles (`tile_Sx`), on the exposures (`exp_Sx`),
+`setools` (`Se`), and the Psf model (`Psf`).  
+
 The following plots show an example of a single CCD, in the center of the focal plane.
 
 | Size-magnitude plot | Star magnitude histogram | Stars in CCD (mag) | Stars in CCD (size) |
 | --- | --- | --- | --- |
-| <img width="250" src="size_mag-2113737-10.png" title="Size-magnitude plot with star selection"> | <img width="250" src="hist_mag_stars-2113737-10.png" title="Magnitude histogram of selected stars"> | <img width="250" src="mag_star_field-2113737-10.png" title="Magnitude distribution in CCD"> | <img width="250" src="fwhm_field-2113737-10.png" title="Size distribution in CCD"> |
+| <img width="250" src="./images/size_mag-2113737-10.png" title="Size-magnitude plot with star selection"> | <img width="250" src="./images/hist_mag_stars-2113737-10.png" title="Magnitude histogram of selected stars"> | <img width="250" src="./images/mag_star_field-2113737-10.png" title="Magnitude distribution in CCD"> | <img width="250" src="./images/fwhm_field-2113737-10.png" title="Size distribution in CCD"> |
 | The stellar locus is well-defined | Magnitude distribution looks reasonable | Stars are relatively homogeneously distributed over the CCD | The uniform and small seeing of CFHT is evident |
 
 To contrast the last plot, here is the case of the CCD in the lower right corner, which shows a known (but yet unexplained) lack of stars
 in the lower parts:
 
-<img width="250" src="fwhm_field-2113737-35.png" title="Size distribution in CCD">
+<img width="250" src="./images/fwhm_field-2113737-35.png" title="Size distribution in CCD">
 
 The statistics output file for the center CCD #10:
 ```bash
-(shapepipe)  dap ~/ShapePipeRun $ cat output/shapepipe_run_2020-03-05_10-00-26/setools_runner/output/stat/star_stat-2113737-10.txt
+cat star_stat-2113737-10.txt
 # Statistics
 Nb objects full cat = 1267
 Nb stars = 160
@@ -260,7 +273,7 @@ to create histograms (as `.txt` tables and `.png` plots) in the directory `stats
 
 | Non-masked objects per CCD | Stars per CCD | FWHM mode |
 | --- | --- | --- |
-| <img width="250" src="1_nb_nonmasked.png" title="Number of non-masked objects per CCD"> | <img width="250" src="2_nb_stars.png" title="Number stars per CCD"> | <img width="250" src="5_mode_fhwm_star.png" title="FWHM mode"> |
+| <img width="250" src="./images/1_nb_nonmasked.png" title="Number of non-masked objects per CCD"> | <img width="250" src="./images/2_nb_stars.png" title="Number stars per CCD"> | <img width="250" src="./images/5_mode_fhwm_star.png" title="FWHM mode"> |
 | No CCD with a very large masked area | No CCD with insufficient stars | Rather broad seeing distribution |
 
 Note that `stats_global` read all `SETool` output stats files found in a given input directory tree. It can thus produce histogram combining
@@ -281,6 +294,13 @@ is computed by the `spread_model` module. Finally, postage stamps
 around galaxies of single-exposure data is extracted with another call
 to `vignetmaker`.
 
+The output directory is
+- `run_sp_MiViSmVi` if the PSF model is `mccd`;
+- `run_sp_tile_PsViSmVi` for the `PSFEx` PSF model.
+
+This corresponds to the MCCD/PSFex interpolation (`Mi`/`Pi`), `vignetmaker` (`Vi`), `spread_model` (`Sm`), and the
+second call to `vignetmaker` (`Vi`).
+
 
 ## Shape measurement
 
@@ -291,20 +311,31 @@ job_sp TILE_ID -j 32
 computes galaxy shapes using the multi-epoch model-fitting method `ngmix`. At the same time,
 shapes of artifically sheared galaxies are obtained for metacalibration.
 
+Shape measurement is performed in parallel for each tile, the number of processes can be specified
+by the user with the option `--nsh_jobs NJOB`. This creates `NJOB` output directories `run_sp_tile_ngmix_Ng<X>u`.
+with `X` = 1 ... `NJOB` containing the result of `ngmix`.
+
 
 ## Paste catalogues
 
-The last real processing step,
+The last real processing step is
 ```bash
 job_sp TILE_ID -j 64
 ```
-which pastes previously obtained information into a _final_ shape catalogue via `make_cat`.
-This includes galaxy detection and basic measurement parameters, the PSF model at
+This task first merges the `NJOB` parallel `ngmix` output files from the previous step into
+one output file. Then, previously obtained information are pasted into a _final_ shape catalogue via `make_cat`.
+Included are galaxy detection and basic measurement parameters, the PSF model at
 galaxy positions, the spread-model classification, and the shape measurement.
 
+Two output directories are created. 
+The first one is `run_sp_Ms` for the `merge_sep` run.
+The second is `run_sp_Mc` for the `make_cat` task; the name is the same for both the `MCCD` and `PSFEx` PSF model.
+
+
 ## Upload results
 
 Optionally, after the pipeline is finished, results can be uploaded to VOspace via
 ```bash
 job_sp TILE_ID -j 128
 ```
+

docs/source/post_processing.md

@@ -1,9 +1,16 @@
 # Post-processing
 
-This page shows all required steps of post-processing a `ShapePipe` run. Some details pertain specifically to
-runs carried out on [canfar](./canfar.md), but most are general.
+This page shows all required steps of post-processing the results from one or
+more `ShapePipe` runs. Post-processing combines various individual `ShapePipe`
+output files, and creates joint results, for example combining individual tile
+catalogues in a large sky area. The output of post-processing is a joint _shape
+catalogue_, containing all required information to create a calibrated shear
+catalogue via _metacalibration_), a joint star catalogue, and PSF diagnostic plots.
 
-1. Retrieve files with `ShapePipe` results
+Some of the following steps pertain specifically to runs carried out on [canfar](https://www.canfar.net/en),
+but most are general.
+
+1. Retrieve `ShapePipe` result files
 
    For a local run on the same machine as for post-processing, nothing needs to be done.
    In some cases, the run was carried out on a remote machine or cluster, and the resulting `ShapePipe`
@@ -50,7 +57,7 @@ runs carried out on [canfar](./canfar.md), but most are general.
       ```
       On success, `ShapePipe` output `fits` and `log` files will be now in various subdirs of the `output` directory.
 
-At this step all required `ShapePipe` resulting output files are available in `.`.
+At this step all required `ShapePipe` resulting output files are available in the current working directory.
 
 2. Optional: Split output in sub-samples
 
@@ -59,7 +66,7 @@ At this step all required `ShapePipe` resulting output files are available in `.
    the previous step. For example, to create the subdir `tiles_W3` with links to result files to `all` for
    those tiles contained in the list `tiles_W3.txt`, do:
    ```bash
-    $SP_ROOT/scripts/python/create_sample_results.py --input_IDs tiles_W3.txt -i . all -o tiles_W3 -v
+    create_sample_results --input_IDs tiles_W3.txt -i . all -o tiles_W3 -v
     ```
     The following steps will then be done in the directory `tiles_W3`.
 
@@ -69,20 +76,27 @@ At this step all required `ShapePipe` resulting output files are available in `.
    ```bash
    post_proc_sp -p PSF
    ```
-   to automatically perform a number of post-processing steps. Chose the PSF model with the option `-p psfex|mccd`. In detail, these are (and can also be
-   done individually
+   to automatically perform a number of post-processing steps. Chose the PSF model with the option
+   `-p psfex|mccd`. In detail, these are (and can also be done individually
    by hand):
 
    A. Analyse psf validation files
 
       ```bash
-      psf_residuals
+      psf_residuals -p PSF
       ```
       with options as for `post_proc_sp`.
       This script identifies all psf validation files (from all processed tiles downloaded to `pwd`), creates symbolic links,
       merges the catalogues, and creates plots of PSF ellipticity, size, and residuals over the focal plane.
 
-   B. Prepare output directory
+   B. Create plots of the PSF and their residuals in the focal plane, as a diagnostic of the overall PSF model.
+     As a scale-dependend test, which propagates directly to the shear correlation function, the rho statistics are computed,
+     see {cite:p}`rowe:10` and {cite:p}`jarvis:16`,
+      ```bash
+      shapepipe_run -c /path/to/shapepipe/example/cfis/config_MsPl_PSF.ini
+      ``` 
+
+   C. Prepare output directory
 
       Create links to all 'final_cat' result files with 
       ```bash
@@ -91,11 +105,13 @@ At this step all required `ShapePipe` resulting output files are available in `.
       The corresponding output directory that is created is `output/run_sp_combined/make_catalog_runner/output`.
       On success, it contains links to all `final_cat` output catalogues
 
-   C. Merge final output files
+   D. Merge final output files
 
-      Create a single main catalog:
+      Create a single main shape catalog:
       ```bash
       merge_final_cat -i <input_dir> -p <param_file> -v
       ```
-      Chose as input directory `input_dir` the output of step B. A default parameter file <param_file> is `/path/to/shapepipe/example/cfis/final_cat.param`. 
-      On success, the file `./final_cat.npy` is created. Depending on the number of input tiles, this file can be several tens of Gb large. 
+      Choose as input directory `input_dir` the output of step C. A default
+      parameter file `<param_file>` is `/path/to/shapepipe/example/cfis/final_cat.param`. 
+      On success, the file `./final_cat.npy` is created. Depending on the number of
+      input tiles, this file can be several tens of Gb large. 

docs/source/refs.bib

@@ -437,6 +437,47 @@ @ARTICLE{jarvis:04
       adsnote = {Provided by the SAO/NASA Astrophysics Data System}
 }
 
+@ARTICLE{jarvis:16, 
+   author = {{Jarvis}, M. and {Sheldon}, E. and {Zuntz}, J. and {Kacprzak}, T. and 
+        {Bridle}, S.~L. and {Amara}, A. and {Armstrong}, R. and {Becker}, M.~R. and
+        {Bernstein}, G.~M. and {Bonnett}, C. and {Chang}, C. and {Das}, R. and
+        {Dietrich}, J.~P. and {Drlica-Wagner}, A. and {Eifler}, T.~F. and 
+        {Gangkofner}, C. and {Gruen}, D. and {Hirsch}, M. and {Huff}, E.~M. and
+        {Jain}, B. and {Kent}, S. and {Kirk}, D. and {MacCrann}, N. and 
+        {Melchior}, P. and {Plazas}, A.~A. and {Refregier}, A. and {Rowe}, B. and
+        {Rykoff}, E.~S. and {Samuroff}, S. and {S{\'a}nchez}, C. and 
+        {Suchyta}, E. and {Troxel}, M.~A. and {Vikram}, V. and {Abbott}, T. and 
+        {Abdalla}, F.~B. and {Allam}, S. and {Annis}, J. and {Benoit-L{\'e}vy}, A. and
+        {Bertin}, E. and {Brooks}, D. and {Buckley-Geer}, E. and {Burke}, D.~L. and
+        {Capozzi}, D. and {Carnero Rosell}, A. and {Carrasco Kind}, M. and
+        {Carretero}, J. and {Castander}, F.~J. and {Clampitt}, J. and 
+        {Crocce}, M. and {Cunha}, C.~E. and {D'Andrea}, C.~B. and {da Costa}, L.~N. and
+        {DePoy}, D.~L. and {Desai}, S. and {Diehl}, H.~T. and {Doel}, P. and  
+        {Fausti Neto}, A. and {Flaugher}, B. and {Fosalba}, P. and {Frieman}, J. and
+        {Gaztanaga}, E. and {Gerdes}, D.~W. and {Gruendl}, R.~A. and 
+        {Gutierrez}, G. and {Honscheid}, K. and {James}, D.~J. and {Kuehn}, K. and
+        {Kuropatkin}, N. and {Lahav}, O. and {Li}, T.~S. and {Lima}, M. and 
+        {March}, M. and {Martini}, P. and {Miquel}, R. and {Mohr}, J.~J. and
+        {Neilsen}, E. and {Nord}, B. and {Ogando}, R. and {Reil}, K. and  
+        {Romer}, A.~K. and {Roodman}, A. and {Sako}, M. and {Sanchez}, E. and
+        {Scarpine}, V. and {Schubnell}, M. and {Sevilla-Noarbe}, I. and
+        {Smith}, R.~C. and {Soares-Santos}, M. and {Sobreira}, F. and 
+        {Swanson}, M.~E.~C. and {Tarle}, G. and {Thaler}, J. and {Thomas}, D. and
+        {Walker}, A.~R. and {Wechsler}, R.~H.}, 
+    title = "{The DES Science Verification weak lensing shear catalogues}",
+  journal = {\mnras},
+archivePrefix = "arXiv",
+   eprint = {1507.05603},
+ primaryClass = "astro-ph.IM",
+     year = 2016,
+    month = aug,
+   volume = 460,
+    pages = {2245-2281},
+      doi = {10.1093/mnras/stw990},
+   adsurl = {http://cdsads.u-strasbg.fr/abs/2016MNRAS.460.2245J},
+  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
+}
+
 @ARTICLE{jensen:21,
        author = {{Jensen}, Jaclyn and {Thomas}, Guillaume and {McConnachie}, Alan W. and {Starkenburg}, Else and {Malhan}, Khyati and {Navarro}, Julio and {Martin}, Nicolas and {Famaey}, Benoit and {Ibata}, Rodrigo and {Chapman}, Scott and {Cuillandre}, Jean-Charles and {Gwyn}, Stephen},
         title = "{Uncovering fossils of the distant Milky Way with UNIONS: NGC 5466 and its stellar stream}",
@@ -595,6 +636,24 @@ @ARTICLE{roberts:22
       adsnote = {Provided by the SAO/NASA Astrophysics Data System}
 }
 
+@ARTICLE{rowe:10,
+   author = {{Rowe}, B.},
+    title = "{Improving PSF modelling for weak gravitational lensing using new methods in model selection}",
+  journal = {\mnras},
+archivePrefix = "arXiv",
+   eprint = {0904.3056},
+ primaryClass = "astro-ph.CO",
+ keywords = {gravitational lensing, methods: data analysis, methods: statistical, cosmology: observations, large-scale structure of Universe},
+     year = 2010,
+    OPTmonth = may,
+   volume = 404,
+    pages = {350-366},
+      doi = {10.1111/j.1365-2966.2010.16277.x},
+   adsurl = {http://adsabs.harvard.edu/abs/2010MNRAS.404..350R},
+  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
+}
+
+
 @ARTICLE{rowe:15,
        author = {{Rowe}, B.~T.~P. and {Jarvis}, M. and {Mandelbaum}, R. and {Bernstein}, G.~M. and {Bosch}, J. and {Simet}, M. and {Meyers}, J.~E. and {Kacprzak}, T. and {Nakajima}, R. and {Zuntz}, J. and {Miyatake}, H. and {Dietrich}, J.~P. and {Armstrong}, R. and {Melchior}, P. and {Gill}, M.~S.~S.},
         title = "{GALSIM: The modular galaxy image simulation toolkit}",

shapepipe/modules/get_images_package/get_images.py

@@ -310,7 +310,7 @@ def retrieve_one(self, in_path, out_path):
 
             log_cmd = ' '.join(sys.argv)
             vcp = vosHandler('vcp')
-            print(log_cmd)
+            self._w_log.info(log_cmd)
 
             attempt = 0
             while attempt < self._n_try: