diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index 33103f9a2..6b075f24d 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -73,20 +73,20 @@ Naming and numbering of the input files can closely follow the original image na files with user-defined summary statistics for each CCD, for example the number of selected stars, or mean and standard deviation of their FWHM. Example: `star_stat-2366993-18.txt` -- Tile ID list +- Tile ID list ASCII file with a tile number on each line. Used for the `get_image_runner` module to download CFIS images (see [Download tiles](#download-tiles)). - - Single-exposure name list - ASCII file with a single-exposure name on each line. Produced by the `find_exposure_runner` module to identify single exposures that were used to create - a given tile. See [Find exposures](#find-exposures)). +- Single-exposure name list + ASCII file with a single-exposure name on each line. Produced by the `find_exposure_runner` module to identify single exposures that were used to create + a given tile. See [Find exposures](#find-exposures)). -- Plots +- Plots The `SETools` module can also produce plots of the objects properties that were selected for a given CCD. The type of plot (histogram, scatter plot, ...) and quantities to plot as well as plot decorations can be specified in the selection criteria config file (see [Select stars](#select-stars)). Example: `hist_mag_stars-2104133-5.png` -- Log files +- Log files The pipeline core and all called modules write ASCII log files to disk. Examples: `process-2366993-6.log`, `log_sp_exp.log`. @@ -125,7 +125,7 @@ job_sp -h ``` for all options. -This script created the subdirectory `$SP_RUN/output` to store all pipeline outputs +This script creates the subdirectory `$SP_RUN/output` to store all pipeline outputs (log files, diagnostics, statistics, output images, catalogues, single-exposure headers with WCS information). Optionally, the subdir `output_star_cat` is created by the used to store the external star catalogues for masking. This is only necessary if the pipeline is run on a cluster without internet connection to access star catalogues. In that case, the star catalogues need to be retrieved outside the pipeline, for example on a login node, and copied to `output_star_cat`. @@ -173,6 +173,9 @@ For the retrieval method the user can choose betwen Note that internet access is required for this step if the download method is `vos`. +An output directory `run_sp_GitFeGie` (in `output`) is created containing the results of `get_images` for tiles (`Git`), +`find_exposures` (`Fe`), and `get_images` for exposures (`Gie`). + ## Prepare input images With @@ -183,6 +186,9 @@ the compressed tile weight image is uncompressed via the `uncompress_fits` modul (one FITS file per CCD) with `split_exp`. Finally, the headers of all single-exposure single-CCD files are merged into a single `sqlite` file, to store the WCS information of the input exposures. +Two output directories are created, `run_sp_Uz` for `uncompress_fits`, and `run_sp_exp_SpMh` for the output of the modules +`split_exp` (`Sp`) and `merge_headers` (`Mh`). + ## Mask images Run @@ -193,14 +199,17 @@ to mask tile and single-exposure single-CCD images. Both tasks are performed by Note that internet access is required for this step, since a reference star catalogue is downloaded. +The output of both masking runs are stored in the output directory `run_sp_MaMa`, with run 1 (2) of +`mask` corresponding to tiles (exposures). + **Diagnostics:** Open a single-exposure single-CCD image and the corresponding pipeline flag in `ds9`, and display both frames next to each other. Example ```bash -ds9 output/shapepipe_run_2020-03-03_15-31-00/split_exp_runner/output/image-2113737-10.fits output/shapepipe_run_2020-03-03_17-29-34/mask_runner/output/pipeline_flag-2113737-10.fits +ds9 image-2113737-10.fits pipeline_flag-2113737-10.fits ``` Choose `zoom fit` for both frames, click `scale zscale` for the image, and `color aips0` for the flag, to display something like this: - + By eye the correspondence between the different flag types and the image can be seen. Note that the two frames might not match perfectly, since (a) WCS @@ -223,21 +232,25 @@ Next, the following tasks are run on the single-exposure single-CCD images: via a call to `psfex_interp`. For MCCD, the modules `merge_starcat`, `mccd_plots`, and `mccd_interp` are called. +The output directory for both the `mccd` and `psfex` options is `run_sp_tile_Sx_exp_SxSePsf`. +This stores the output of SExtractor on the tiles (`tile_Sx`), on the exposures (`exp_Sx`), +`setools` (`Se`), and the Psf model (`Psf`). + The following plots show an example of a single CCD, in the center of the focal plane. | Size-magnitude plot | Star magnitude histogram | Stars in CCD (mag) | Stars in CCD (size) | | --- | --- | --- | --- | -| | | | | +| | | | | | The stellar locus is well-defined | Magnitude distribution looks reasonable | Stars are relatively homogeneously distributed over the CCD | The uniform and small seeing of CFHT is evident | To contrast the last plot, here is the case of the CCD in the lower right corner, which shows a known (but yet unexplained) lack of stars in the lower parts: - + The statistics output file for the center CCD #10: ```bash -(shapepipe) dap ~/ShapePipeRun $ cat output/shapepipe_run_2020-03-05_10-00-26/setools_runner/output/stat/star_stat-2113737-10.txt +cat star_stat-2113737-10.txt # Statistics Nb objects full cat = 1267 Nb stars = 160 @@ -260,7 +273,7 @@ to create histograms (as `.txt` tables and `.png` plots) in the directory `stats | Non-masked objects per CCD | Stars per CCD | FWHM mode | | --- | --- | --- | -| | | | +| | | | | No CCD with a very large masked area | No CCD with insufficient stars | Rather broad seeing distribution | Note that `stats_global` read all `SETool` output stats files found in a given input directory tree. It can thus produce histogram combining @@ -281,6 +294,13 @@ is computed by the `spread_model` module. Finally, postage stamps around galaxies of single-exposure data is extracted with another call to `vignetmaker`. +The output directory is +- `run_sp_MiViSmVi` if the PSF model is `mccd`; +- `run_sp_tile_PsViSmVi` for the `PSFEx` PSF model. + +This corresponds to the MCCD/PSFex interpolation (`Mi`/`Pi`), `vignetmaker` (`Vi`), `spread_model` (`Sm`), and the +second call to `vignetmaker` (`Vi`). + ## Shape measurement @@ -291,20 +311,31 @@ job_sp TILE_ID -j 32 computes galaxy shapes using the multi-epoch model-fitting method `ngmix`. At the same time, shapes of artifically sheared galaxies are obtained for metacalibration. +Shape measurement is performed in parallel for each tile, the number of processes can be specified +by the user with the option `--nsh_jobs NJOB`. This creates `NJOB` output directories `run_sp_tile_ngmix_Ngu`. +with `X` = 1 ... `NJOB` containing the result of `ngmix`. + ## Paste catalogues -The last real processing step, +The last real processing step is ```bash job_sp TILE_ID -j 64 ``` -which pastes previously obtained information into a _final_ shape catalogue via `make_cat`. -This includes galaxy detection and basic measurement parameters, the PSF model at +This task first merges the `NJOB` parallel `ngmix` output files from the previous step into +one output file. Then, previously obtained information are pasted into a _final_ shape catalogue via `make_cat`. +Included are galaxy detection and basic measurement parameters, the PSF model at galaxy positions, the spread-model classification, and the shape measurement. +Two output directories are created. +The first one is `run_sp_Ms` for the `merge_sep` run. +The second is `run_sp_Mc` for the `make_cat` task; the name is the same for both the `MCCD` and `PSFEx` PSF model. + + ## Upload results Optionally, after the pipeline is finished, results can be uploaded to VOspace via ```bash job_sp TILE_ID -j 128 ``` + diff --git a/docs/source/post_processing.md b/docs/source/post_processing.md index 25dee7180..d3edec0ea 100644 --- a/docs/source/post_processing.md +++ b/docs/source/post_processing.md @@ -1,9 +1,16 @@ # Post-processing -This page shows all required steps of post-processing a `ShapePipe` run. Some details pertain specifically to -runs carried out on [canfar](./canfar.md), but most are general. +This page shows all required steps of post-processing the results from one or +more `ShapePipe` runs. Post-processing combines various individual `ShapePipe` +output files, and creates joint results, for example combining individual tile +catalogues in a large sky area. The output of post-processing is a joint _shape +catalogue_, containing all required information to create a calibrated shear +catalogue via _metacalibration_), a joint star catalogue, and PSF diagnostic plots. -1. Retrieve files with `ShapePipe` results +Some of the following steps pertain specifically to runs carried out on [canfar](https://www.canfar.net/en), +but most are general. + +1. Retrieve `ShapePipe` result files For a local run on the same machine as for post-processing, nothing needs to be done. In some cases, the run was carried out on a remote machine or cluster, and the resulting `ShapePipe` @@ -50,7 +57,7 @@ runs carried out on [canfar](./canfar.md), but most are general. ``` On success, `ShapePipe` output `fits` and `log` files will be now in various subdirs of the `output` directory. -At this step all required `ShapePipe` resulting output files are available in `.`. +At this step all required `ShapePipe` resulting output files are available in the current working directory. 2. Optional: Split output in sub-samples @@ -59,7 +66,7 @@ At this step all required `ShapePipe` resulting output files are available in `. the previous step. For example, to create the subdir `tiles_W3` with links to result files to `all` for those tiles contained in the list `tiles_W3.txt`, do: ```bash - $SP_ROOT/scripts/python/create_sample_results.py --input_IDs tiles_W3.txt -i . all -o tiles_W3 -v + create_sample_results --input_IDs tiles_W3.txt -i . all -o tiles_W3 -v ``` The following steps will then be done in the directory `tiles_W3`. @@ -69,20 +76,27 @@ At this step all required `ShapePipe` resulting output files are available in `. ```bash post_proc_sp -p PSF ``` - to automatically perform a number of post-processing steps. Chose the PSF model with the option `-p psfex|mccd`. In detail, these are (and can also be - done individually + to automatically perform a number of post-processing steps. Chose the PSF model with the option + `-p psfex|mccd`. In detail, these are (and can also be done individually by hand): A. Analyse psf validation files ```bash - psf_residuals + psf_residuals -p PSF ``` with options as for `post_proc_sp`. This script identifies all psf validation files (from all processed tiles downloaded to `pwd`), creates symbolic links, merges the catalogues, and creates plots of PSF ellipticity, size, and residuals over the focal plane. - B. Prepare output directory + B. Create plots of the PSF and their residuals in the focal plane, as a diagnostic of the overall PSF model. + As a scale-dependend test, which propagates directly to the shear correlation function, the rho statistics are computed, + see {cite:p}`rowe:10` and {cite:p}`jarvis:16`, + ```bash + shapepipe_run -c /path/to/shapepipe/example/cfis/config_MsPl_PSF.ini + ``` + + C. Prepare output directory Create links to all 'final_cat' result files with ```bash @@ -91,11 +105,13 @@ At this step all required `ShapePipe` resulting output files are available in `. The corresponding output directory that is created is `output/run_sp_combined/make_catalog_runner/output`. On success, it contains links to all `final_cat` output catalogues - C. Merge final output files + D. Merge final output files - Create a single main catalog: + Create a single main shape catalog: ```bash merge_final_cat -i -p -v ``` - Chose as input directory `input_dir` the output of step B. A default parameter file is `/path/to/shapepipe/example/cfis/final_cat.param`. - On success, the file `./final_cat.npy` is created. Depending on the number of input tiles, this file can be several tens of Gb large. + Choose as input directory `input_dir` the output of step C. A default + parameter file `` is `/path/to/shapepipe/example/cfis/final_cat.param`. + On success, the file `./final_cat.npy` is created. Depending on the number of + input tiles, this file can be several tens of Gb large. diff --git a/docs/source/refs.bib b/docs/source/refs.bib index 5adbc83f0..f2f5c933d 100644 --- a/docs/source/refs.bib +++ b/docs/source/refs.bib @@ -437,6 +437,47 @@ @ARTICLE{jarvis:04 adsnote = {Provided by the SAO/NASA Astrophysics Data System} } +@ARTICLE{jarvis:16, + author = {{Jarvis}, M. and {Sheldon}, E. and {Zuntz}, J. and {Kacprzak}, T. and + {Bridle}, S.~L. and {Amara}, A. and {Armstrong}, R. and {Becker}, M.~R. and + {Bernstein}, G.~M. and {Bonnett}, C. and {Chang}, C. and {Das}, R. and + {Dietrich}, J.~P. and {Drlica-Wagner}, A. and {Eifler}, T.~F. and + {Gangkofner}, C. and {Gruen}, D. and {Hirsch}, M. and {Huff}, E.~M. and + {Jain}, B. and {Kent}, S. and {Kirk}, D. and {MacCrann}, N. and + {Melchior}, P. and {Plazas}, A.~A. and {Refregier}, A. and {Rowe}, B. and + {Rykoff}, E.~S. and {Samuroff}, S. and {S{\'a}nchez}, C. and + {Suchyta}, E. and {Troxel}, M.~A. and {Vikram}, V. and {Abbott}, T. and + {Abdalla}, F.~B. and {Allam}, S. and {Annis}, J. and {Benoit-L{\'e}vy}, A. and + {Bertin}, E. and {Brooks}, D. and {Buckley-Geer}, E. and {Burke}, D.~L. and + {Capozzi}, D. and {Carnero Rosell}, A. and {Carrasco Kind}, M. and + {Carretero}, J. and {Castander}, F.~J. and {Clampitt}, J. and + {Crocce}, M. and {Cunha}, C.~E. and {D'Andrea}, C.~B. and {da Costa}, L.~N. and + {DePoy}, D.~L. and {Desai}, S. and {Diehl}, H.~T. and {Doel}, P. and + {Fausti Neto}, A. and {Flaugher}, B. and {Fosalba}, P. and {Frieman}, J. and + {Gaztanaga}, E. and {Gerdes}, D.~W. and {Gruendl}, R.~A. and + {Gutierrez}, G. and {Honscheid}, K. and {James}, D.~J. and {Kuehn}, K. and + {Kuropatkin}, N. and {Lahav}, O. and {Li}, T.~S. and {Lima}, M. and + {March}, M. and {Martini}, P. and {Miquel}, R. and {Mohr}, J.~J. and + {Neilsen}, E. and {Nord}, B. and {Ogando}, R. and {Reil}, K. and + {Romer}, A.~K. and {Roodman}, A. and {Sako}, M. and {Sanchez}, E. and + {Scarpine}, V. and {Schubnell}, M. and {Sevilla-Noarbe}, I. and + {Smith}, R.~C. and {Soares-Santos}, M. and {Sobreira}, F. and + {Swanson}, M.~E.~C. and {Tarle}, G. and {Thaler}, J. and {Thomas}, D. and + {Walker}, A.~R. and {Wechsler}, R.~H.}, + title = "{The DES Science Verification weak lensing shear catalogues}", + journal = {\mnras}, +archivePrefix = "arXiv", + eprint = {1507.05603}, + primaryClass = "astro-ph.IM", + year = 2016, + month = aug, + volume = 460, + pages = {2245-2281}, + doi = {10.1093/mnras/stw990}, + adsurl = {http://cdsads.u-strasbg.fr/abs/2016MNRAS.460.2245J}, + adsnote = {Provided by the SAO/NASA Astrophysics Data System} +} + @ARTICLE{jensen:21, author = {{Jensen}, Jaclyn and {Thomas}, Guillaume and {McConnachie}, Alan W. and {Starkenburg}, Else and {Malhan}, Khyati and {Navarro}, Julio and {Martin}, Nicolas and {Famaey}, Benoit and {Ibata}, Rodrigo and {Chapman}, Scott and {Cuillandre}, Jean-Charles and {Gwyn}, Stephen}, title = "{Uncovering fossils of the distant Milky Way with UNIONS: NGC 5466 and its stellar stream}", @@ -595,6 +636,24 @@ @ARTICLE{roberts:22 adsnote = {Provided by the SAO/NASA Astrophysics Data System} } +@ARTICLE{rowe:10, + author = {{Rowe}, B.}, + title = "{Improving PSF modelling for weak gravitational lensing using new methods in model selection}", + journal = {\mnras}, +archivePrefix = "arXiv", + eprint = {0904.3056}, + primaryClass = "astro-ph.CO", + keywords = {gravitational lensing, methods: data analysis, methods: statistical, cosmology: observations, large-scale structure of Universe}, + year = 2010, + OPTmonth = may, + volume = 404, + pages = {350-366}, + doi = {10.1111/j.1365-2966.2010.16277.x}, + adsurl = {http://adsabs.harvard.edu/abs/2010MNRAS.404..350R}, + adsnote = {Provided by the SAO/NASA Astrophysics Data System} +} + + @ARTICLE{rowe:15, author = {{Rowe}, B.~T.~P. and {Jarvis}, M. and {Mandelbaum}, R. and {Bernstein}, G.~M. and {Bosch}, J. and {Simet}, M. and {Meyers}, J.~E. and {Kacprzak}, T. and {Nakajima}, R. and {Zuntz}, J. and {Miyatake}, H. and {Dietrich}, J.~P. and {Armstrong}, R. and {Melchior}, P. and {Gill}, M.~S.~S.}, title = "{GALSIM: The modular galaxy image simulation toolkit}", diff --git a/shapepipe/modules/get_images_package/get_images.py b/shapepipe/modules/get_images_package/get_images.py index 0cd748d60..52eae63b0 100644 --- a/shapepipe/modules/get_images_package/get_images.py +++ b/shapepipe/modules/get_images_package/get_images.py @@ -310,7 +310,7 @@ def retrieve_one(self, in_path, out_path): log_cmd = ' '.join(sys.argv) vcp = vosHandler('vcp') - print(log_cmd) + self._w_log.info(log_cmd) attempt = 0 while attempt < self._n_try: diff --git a/shapepipe/modules/mccd_package/__init__.py b/shapepipe/modules/mccd_package/__init__.py index a748f49f0..e42975319 100644 --- a/shapepipe/modules/mccd_package/__init__.py +++ b/shapepipe/modules/mccd_package/__init__.py @@ -143,11 +143,18 @@ :Output: Several plots -This module generates different plots of the input merged star catalogue. -The plots include mean shapes, where the ellipticity error is plotted as a -function of the focal plane positions. The errors are binned through all the -merged catalogue in the grid defined by ``X_GRID`` and ``Y_GRID``. The other -plots are a histogram of the ellipticity errors and the rho stats plot. +This module generates different series of plots of the input merged star catalogue. + +One series of plots are the _mean shapes_, which is the PSF ellipticity and size +and their residuals as a function of the focal plane positions. Values and residuals +are combined from all merged catalogues, and binned in the focal plane according to +``X_GRID`` and ``Y_GRID``. + +Another series of plots are a histogram of the ellipticity errors. + +A third series are the rho statistics plot, which are correlation functions of +various combinations of PSF ellipticity, size, and their residuals, +see :cite:`rowe:10` and :cite:`jarvis:16`. Module-specific config file entries ----------------------------------- @@ -155,15 +162,15 @@ PSF: str PSF model type, options are ``mccd`` or ``psfex`` PLOT_MEANSHAPES: bool - Option to produce mean shapes plots + Option to produce _mean shapes_ plots X_GRID: int - Number of bins in the X axis of one CCD for the mean shapes plot grid + Number of bins in the X axis of one CCD for the _mean shapes_ plot grid Y_GRID: int - Number of bins in the Y axis of one CCD for the mean shapes plot grid + Number of bins in the Y axis of one CCD for the _mean shapes_ plot grid MAX_E: float - Max value for ellipticity in the mean shapes plot + Max value for ellipticity in the _mean shapes_ plot MAX_DE: float - Max value for the residual ellipticity in the mean shapes plot + Max value for the residual ellipticity in the _mean shapes_ plot PLOT_HISTOGRAMS: bool Option to produce histogram plots for shape errors REMOVE_OUTLIERS: bool