From e6054941155f7c3b9db79f4eaa5e7224d17c13f0 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Wed, 2 Mar 2022 15:16:18 +0100 Subject: [PATCH 01/27] updating job script and tutorial --- docs/source/pipeline_tutorial.md | 18 ++++++------------ scripts/sh/job_sp.bash | 14 ++------------ 2 files changed, 8 insertions(+), 24 deletions(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index fc3bc1a91..cfb1d07c8 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -34,16 +34,11 @@ ## Quick start -Run the pipeline on a single example CFIS image: -1. Download and un-tar the - [example data](https://drive.google.com/file/d/1AowB2_XJtoOI8eB21cKeQhhh3_8mDEHw/view?usp=sharing). - The file size is 4.1 Gb, this might take a while. This data consists of a single CFIS tile image and weight, and all contributing exposure - images, weights, and flag FITS files. - Note that these files represent unpublished, sensitive pixel data, which according to [CFIS rules](https://www.cfht.hawaii.edu/Science/CFIS/cfisaccess.html) should not be downloaded outside of France or Canada, and not be shared with non-CFIS members. -2. [Install](https://github.com/cosmostat/shapepipe#installing-shapepipe) `ShapePipe` and activate the `shapepipe` conda environment. +Run the entire pipeline on a single example CFIS image with tile ID 246.290: +1. [Install](https://github.com/cosmostat/shapepipe#installing-shapepipe) `ShapePipe` and activate the `shapepipe` conda environment. 3. Run the job script ```bash -job_sp -r symlink 246.290 +job_sp 246.290 -j 127 ``` ## Introduction @@ -124,10 +119,9 @@ Naming and numbering of the input files can closely follow the original image na ### CFIS processing -`ShapePipe` splits the processing of CFIS images into three parts: -1.) [Retrieve and prepare input images](#retrieve-and-prepare-input-images) -2.) [Process single exposure images](#process-single-exposure-images) -3.) [Process stacked images](#process-stacked-images) +`ShapePipe` splits the processing of CFIS images into several parts: +These are the retrieval and preparation of input images, processing of single exposures, +processing of tile images, creation and upload (optional) of final shear catalogues. The following flowchart visualised the processing parts and steps. diff --git a/scripts/sh/job_sp.bash b/scripts/sh/job_sp.bash index 5406fb4cf..8679403c2 100755 --- a/scripts/sh/job_sp.bash +++ b/scripts/sh/job_sp.bash @@ -21,7 +21,6 @@ fi # Command line arguments ## Default values -do_env=0 job=255 #config_dir='vos:cfis/cosmostat/kilbinger/cfis' config_dir=$VM_HOME/shapepipe/example/cfis @@ -41,8 +40,8 @@ usage="Usage: $(basename "$0") [OPTIONS] TILE_ID_1 [TILE_ID_2 [...]] \t 1: retrieve images (online if method=vos)\n \t 2: prepare images (offline)\n \t 4: mask (online)\n - \t 8: processing of stars on exposures (offline)\n - \t 16: detection and vignets (offline)\n + \t 8: detection of galaxies on tiles; processing of stars on exposures (offline)\n + \t 16: galaxy selection on tiles (offline)\n \t 32: shapes and morphology (offline)\n \t 64: paste catalogues (offline)\n \t 128: upload results (online)\n @@ -80,9 +79,6 @@ while [ $# -gt 0 ]; do echo -ne $usage exit 0 ;; - -e) - do_env=1 - ;; -j|--job) job="$2" shift @@ -280,12 +276,6 @@ function print_env() { ### Start ### -if [ $do_env == 1 ]; then - print_env - echo "Exiting" - return -fi - echo "Start" echo "Processing $n_tile tile(s)" From 9346fb6f34abbdfa40e8d9b9edcee2892b22be38 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Wed, 2 Mar 2022 18:23:20 +0100 Subject: [PATCH 02/27] tutorial; fix config file --- docs/source/pipeline_tutorial.md | 184 ++++++++--------------- example/cfis/config_tile_Sx_exp_mccd.ini | 2 +- 2 files changed, 62 insertions(+), 124 deletions(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index cfb1d07c8..50354bf59 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -129,9 +129,11 @@ The following flowchart visualised the processing parts and steps. Below, the individual processing steps are described in detail. -### Path variables +### Input and output paths -All required variables are automatically set in the job script `job_sp`. If an example config file is to be run outside this script, +All required paths are automatically set in the job script `job_sp`. + +If an example config file is run outside this script, the following path variables might need to be defined. - `$SP_RUN`: Run directory of `ShapePipe`. In general this is just `pwd`, and can be set via ```bash @@ -140,161 +142,81 @@ the following path variables might need to be defined. but on a cluster this directory might be different. - `$SP_CONFIG`: Path to configuration files. In our example this is `$SP_BASE/example/cfis`. +In addition, the output path `$SP_RUN/output` needs to be created by the user before running `ShapePipe`. + ### Running the pipeline -See the main `ShapePipe` readme for more details. +The job script to run the pipeline in its entity or in parts is `job_sp[.bash]`. Type +```bash +job_sp -h +``` +for all options. -In `$SP_RUN` the following subdirectory `output` is created to store all pipeline outputs +This script created the subdirectory `$SP_RUN/output` to store all pipeline outputs (log files, diagnostics, statistics, output images, catalogues, single-exposure headers with WCS information). -Note that this is done automcatically in `job_sp`. Optionally, the subdir `output_star_cat` is created by the used to store the external star catalogues for masking. This is only necessary if the pipeline is run on a cluster without internet connection to access star catalogues. In that case, the star catalogues need to be retrieved outside the pipeline, for example on a login node, and copied to `output_star_cat`. -In general, a call to the pipeline is done as follows, after activating the `shapepipe` conda environment (indicated by `(shapepipe)` at the beginning of the shell prompt. +The job script automaticall performs a number of subsequent calls to the `ShapePipe` executable `shapepipe_run`, as ```bash shapepipe_run -c $SP_CONFIG/.ini ``` The config file `.ini` contains the configuration for one or more modules. +See the main `ShapePipe` readme for more details. - -## Retrieve input images - -Before running `ShapePipe` we need to select the tiles to be processed. +The user specifies which steps are run with the command line option `-j JOB`. The integer value `JOB` +is bit-coded such that arbitrary combinations of steps can be run with a single call to `job_sp`. For +example, to run steps #1 and #2, type `job_sp -j 3`. ### Select tiles -The result of this selection is a text file with a tile IDs on each line, in the form `XYZ.ABC`. If the ID(s) are known in advance, this can be done by hand, e.g. -```bash -echo 002.240 > tile_numbers.txt -``` +To run the job script, one or more CFIS tiles need to be chosen. If the tile IDs are known, they are provided to `job_sp` on the command line. -If this is not the case, tiles can be selected to by sky coordinates, with the script `cfis_field_select`. +If the tile IDs are not known a priori, they can be selected via sky coordinates, with the script `cfis_field_select`. For example, to find the tile number for a Planck cluster at R.A.=213.68 deg, dec=57.79 deg, run: ```bash -cfis_field_select -i $SP_BASE/aux/CFIS/tiles_202007/tiles_all_order.txt --coord 213.68deg_54.79deg -t tile --input_format ID_only --out_name_only --out_ID_only -s -o tile_numbers +cfis_field_select -i /path/to/shapepipe/auxdir/CFIS/tiles_202007/tiles_all_order.txt --coord 213.68deg_54.79deg -t tile --input_format ID_only --out_name_only --out_ID_only -s ``` -The input text file (with `-i`) contains a list of CFIS tiles, this can also be directory containing the tile FITS files. - -To select tiles covering an entire sky area, for example CFHTLS-W3, do: -```bash -cfis_field_select -i /path/to/shapepipe/aux/CFIS/tiles_202007/tiles_all_order.txt --area 208deg_50.75deg_221.deg_58deg -v --input_format ID_only --out_name_only --out_ID_only -s -o tile_numbers --plot -``` - -Now we are ready to run the first, pre-processing `ShapePipe` modules. - -### Retrieve tiles - -The module section `[GET_IMAGES_RUNNER]` first contains the input tile ID list file path (default is `tile_numbers.txt`). This is assembled from `FILE_PATTERN`, `FILE_EXT`, and `NUMBERING_SCHEME`. Since there -is only a single input text file with no number, the latter entry is empty. See `File options` in the [configuration](configuration.md) for more details on the numbering scheme. - -Next, we need to specify for the images input path (for example a VOS url), -input file pattern and their extension. The entry `INPUT_FILE_PATTERN` includes the -tile ID(s) as dummy template, similarly to the `NUMBERING_SCHEME`. In -addition, the input numbering scheme as python `regexp` needs to be specified -by `INPUT_NUMBERING`, which matches the tile number in the tile number list. To -summarize, `INPUT_FILE_PATTERN` needs to match the file name to be -downloaded/linked. `INPUT_NUMBERING` needs to match the ID numbers in the tile -ID list text file `tile_numbers.txt`. - -Next, the output file pattern without the tile number is specified in the config file. Here this needs to -be different from the original image names to be consistent with the `ShapePipe` naming conventions, e.g.~no -dots other than for the file extension. - -### Uncompress tile weights - -The compressed stack weights (.fits.fz/.fitsfz files) need to be uncompressed before further processing. An example config file is `$SP_CONFIG/config_unfz_w.ini`. Except from the input file pattern and extension (see `File Options` in [configuration](configuration.md), we need to specify the output file patern (can be the same as the input), and the HDU number of the weight image data: - -### Find single exposures - +The input text file (provide via the flag `-i`) contains a list of CFIS tiles, this can also be directory containing the tile FITS files. -### Retrieve single exposures -This process works as the one to download tiles, see [Download tiles](#download-tiles). The single-exposure names are read from the output ascii file of the previous module (`find-exposure-runner`). Single-exposure images, weights, and flags are retrieved. Here is the example `$SP_CONFIG/config_tile_Gie.ini` relevant section. +The following sections describe the different steps that are performed with `job_sp`. -On sucess, single-exposure images, weights, and flags are downloaded, or links to existing files are created. - -To perform the pre-processing steps detailed above, we can either run them module by module: -```bash -shapepipe_run -c $SP_CONFIG/config_tile_Git.ini -shapepipe_run -c $SP_CONFIG/config_tile_Uz.ini -shapepipe_run -c $SP_CONFIG/config_tile_Fe.ini -shapepipe_run -c $SP_CONFIG/config_tile_Gie.ini -``` -or in one go: -```bash -shapepipe_run -c $SP_CONFIG/config_tile_GitUzFeGie.ini -``` -This ends the pre-processing, now we can proceed to the actual (parallel) image processing. - -## Process single exposure images +## Retrieve input images -Single exposures can be processed via a single run of `ShapePipe`, which calls all required modules one after another. The corresponding example config file is `$SP_CONFIG/config_exp.ini`, and the command is simply +The command ```bash -shapepipe_run -c $SP_CONFIG/config_exp.ini +job_sp TILE_ID -j 1 ``` -Alternatively each module can be executed by a separate `ShapePipe` call. The corresponding single-module example config files are detailed in the sections below. - -### Split images - -The first step of single-exposure processing is to split the single-exposures images into -files that contain one CCD each. +retrieves the image and weight corresponding to TILE_ID using the module `get_images`. +It then identifies the exposures that were used to create the tile image via the 'find_exposures` runner. +Finally, another call to `get_images` retrieves the exposure images, weights, and flag files. -### Merge WCS headers +For the retrieval method the user can choose betwen +- download from VOspace (`-r vos`); +- create symbolic link to existing file on disk (`-r symlink`). -**Module:** merge_headers_runner -**Parent:** split_exp_runner -**Input:** SQL files with single-exposure header information -**Output:** single SQL file with combined header information +Note that internet access is required for this step if the download method is `vos`. -This pipeline module merges the WCS information (image -transformation and distortions, computed during astrometrical calibration) -for each CCD. At the end, this information has to be merged back into a single file. +## Prepare input images -The example config file is `$SP_CONFIG/config_exp_Mh.ini`: -```ini -[MERGE_HEADERS_RUNNER] -INPUT_DIR = last:split_exp_runner -FILE_PATTERN = headers -FILE_EXT = .npy -NUMBERING_SCHEME = -0000000 +With +```bash +job_sp TILE_ID -j 2 ``` -Since this produces a single output file instead of a file per input image, it is more convenient to have this file in -a separated directory for later use. - -On success, a single `.sqlite` file is created. +the compressed tile weight image is uncompressed via the `uncompress_fits` module. Then, the single-exposure images, weight, and flags are split into single-exposure single-CCD file +(one FITS file per CCD) with `split_exp`. +Finally, the headers of all single-exposure single-CCD files are merged into a single `sqlite` file, to store the WCS information of the input exposures. -> Note: On cc this module failed to run alone. It should be run together with -the previous one, `split_exp_runner`. +## Mask images -### Mask images - -This module creates masks for bright stars, diffraction spikes, Messier objects, -borders, and other artifacts. It joins the newly created mask with the already -existing masks (from the input flag files) of cosmic rays and various artifacts. - -The mask config file -specifies the mask properties for border, halos, spikes, Messier -objects, and external flag input (in our case provided from CFIS pre-processing). - -It points to various default parameter files for the different mask types, -make sure that that paths are correct, in our case -`$SP_CONFIG/mask_default/` in front of each file name. - -This module requires a star catalogue containing position and magnitude -of bright stars. By default this is automatically created by accessing online -star catalogues. - -Since in some cases computing nodes on clusters might not have -internet access, such a catalog can also be created for each image, before running -this module using as follows +Run ```bash -mkdir -o output_star_cat -create_star_cat input_exposures output_star_cat exp +job_sp TILE_ID -j 4 ``` -In this case the star catalogue needs to be specified as input in the config file, -and the config flag ``USE_EXT_STAR`` has to be set to ``True``. -``` -On success, pipeline-flag files are created. +to mask tile and single-exposure single-CCD images. Both tasks are performed by two calls to the `mask` runner. + +Note that internet access is required for this step, since a reference star catalogue is downloaded. **Diagnostics:** Open a single-exposure single-CCD image and the corresponding pipeline flag in `ds9`, and display both frames next to each other. Example @@ -310,6 +232,22 @@ seen. Note that the two frames might not match perfectly, since (a) WCS information is not available in the flag file FITS headers; (b) the image can have a zero-padded pixel border, which is not accounted for by `ds9`. +## Detect objects on tiles; process stars on single exposures + +The call +```bash +job_sp TILE_ID -j 8 +``` +performs a number of steps. First, objects on the tiles are deteced with the `sextractor` runner. +Next, the following tasks are run on the single-exposure single-CCD images: +- Objects are deteced with `sextractor`. +- Star candidates are selected via `setools`. +- The PSF model is created, either with `psfex` for PSFex, or + with `mccd_preprocessing` and `mccd_fit_val` for MCCD. +- The PSF model is interpolated to star positions for validation. For the PSFEx model, this is done + via a call to `psfex_interp`. For MCCD, the modules `merge_starcat`, `mccd_plots`, and + `mccd_interp` are called. + ### Extract sources **Module:** sextractor_runner diff --git a/example/cfis/config_tile_Sx_exp_mccd.ini b/example/cfis/config_tile_Sx_exp_mccd.ini index 4d1d7cb64..2497e7b28 100644 --- a/example/cfis/config_tile_Sx_exp_mccd.ini +++ b/example/cfis/config_tile_Sx_exp_mccd.ini @@ -22,7 +22,7 @@ RUN_NAME = run_sp_SxSePsfSx MODULE = sextractor_runner, setools_runner, mccd_preprocessing_runner, mccd_fit_val_runner, merge_starcat_runner, mccd_plots_runner, - sextractor_runner + mccd_interp_runner, sextractor_runner # Run mode, SMP or MPI MODE = SMP From da029f32c729f5c3f8818a25ea1a6b044a682a15 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Wed, 2 Mar 2022 20:33:39 +0100 Subject: [PATCH 03/27] tutorial finished --- docs/source/pipeline_tutorial.md | 548 +++---------------------------- 1 file changed, 45 insertions(+), 503 deletions(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index 50354bf59..082f4e1c0 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -6,31 +6,19 @@ 1. [Introduction](#introduction) 1. [File types and names](#file-types-and-names) 1. [CFIS processing](#cfis-processing) - 1. [Running the pipeline](#running-the-pipeline) - 1. [Path variables](#path-variables) -1. [Retrieve input images](#retrieve-input-images) + 1. [Input and output paths](#input-and-output-paths) + 1. [Job and pipeline scripts](#job-and-pipeline-scripts) 1. [Select tiles](#select-tiles) - 1. [Retrieve tiles](#retrieve-tiles) - 1. [Uncompress tile weights](#uncompress-tile-weights) - 1. [Find single exposures](#find-single-exposures) - 1. [Retrieve single exposure](#retrieve-single-exposures) -1. [Process single exposure images](#process-single-exposure-images) - 1. [Split images](#split-images) - 1. [Merge WCS headers](#merge-wcs-headers) +1. [Run the pipeline](#run-the-pipeline) + 1. [Retrieve input images](#retrieve-input-images) + 1. [Prepare input images](#prepare-input-images) 1. [Mask images](#mask-images) - 1. [Extract sources](#extract-sources) - 1. [Select stars](#select-stars) - 1. [Create PSF model](#create-psf-model) - 1. [Validation tests](#validation-tests) -1. [Process stacked images](#process-stacked-images) - 1. [Mask stacks](#mask-stacks) - 1. [Extract sources on stacks](#extract-sources-on-stacks) - 1. [Interpolate multi-epoch PSF](#interpolate-multi-epoch-psf) - 1. [Create weight postage stamps](#create-weight-postage-stamps) - 1. [Compute spread model](#compute-spread-model) - 1. [Create single-exposure postage stamps](#create-single-exposure-postage-stamps) - 1. [Multi-epoch shape measurement with ngmix](#multi-epoch-shape-measurement-with-ngmix) - 1. [Create final catalog](#create-final-catalog) + 1. [Detect objects on tiles and process stars on single exposures](#detect-objects-on-tiles-and-process-stars-on-single-exposures) + 1. [Global star sample statistics](#global-star-sample-statistics) + 1. [Galaxy selection](#galaxy-selection) + 1. [Shape measurement](#shape-measurement) + 1. [Paste catalogues](#paste-catalogues) + 1. [Upload results](#upload-results) ## Quick start @@ -144,7 +132,7 @@ the following path variables might need to be defined. In addition, the output path `$SP_RUN/output` needs to be created by the user before running `ShapePipe`. -### Running the pipeline +### Job and pipeline scripts The job script to run the pipeline in its entity or in parts is `job_sp[.bash]`. Type ```bash @@ -182,7 +170,9 @@ The input text file (provide via the flag `-i`) contains a list of CFIS tiles, t The following sections describe the different steps that are performed with `job_sp`. -## Retrieve input images +## Run the pipeline + +### Retrieve input images The command ```bash @@ -232,7 +222,7 @@ seen. Note that the two frames might not match perfectly, since (a) WCS information is not available in the flag file FITS headers; (b) the image can have a zero-padded pixel border, which is not accounted for by `ds9`. -## Detect objects on tiles; process stars on single exposures +## Detect objects on tiles and process stars on single exposures The call ```bash @@ -248,136 +238,19 @@ Next, the following tasks are run on the single-exposure single-CCD images: via a call to `psfex_interp`. For MCCD, the modules `merge_starcat`, `mccd_plots`, and `mccd_interp` are called. -### Extract sources - -**Module:** sextractor_runner -**Parent:** split_exp_runner, mask_runner -**Input:** single-exp_single-CCD image[, weights] [, flags] [, PSF file] [, detection image] [, detection weight] -**Output:** sextractor catalogue - -The purpose of source extraction/source identification on single exposures is -to select stars in the next step. Therefore, a relatively high -detection threshold is chosen to avoid to detect too many low-SNR -artifacts, and to reduce the output catalogue size. The following config entry -is -```ini -DETECT_THRESH 1.5 # or , in mag.arcsec-2 -ANALYSIS_THRESH 1.5 # or , in mag.arcsec-2 -``` -in the file `$SP_CONFIG/default_exp.sex`. - -The module config file `$SP_CONFIG/config_exp_Sx.ini` specifies input modules, file pattern, and numbering scheme. Note that the flag files are the ones created by the [mask module](#mask-images), with name `pipeline_flag`. Next, the path to the `SExtractor` executable (installed in the `shapepipe` conda environment by default) is given, and the `SExtractor` configuration files, see below. - -Then, boolean flags indicate the presence or absence of the additional input images for the `FILE_PATTERN` line. These are a weight image, a flag image, a PSF file, a distinct image for detection (for `SExtractor` in dual-image mode) that is different from the measurement image, and a distinct detection weight. In our case, we don't have a PSF [yet](#create-psf-model), and we do measurement and detection on the same images. - -After that, two entries determine the FITS header key that contains the photometric zero-point (by default 30 if not given). Then, information about the background handling are indicated before the output file suffix. A post processing flag is set to `False` for single-exposures: -```ini -[SEXTRACTOR_RUNNER] -INPUT_MODULE = split_exp_runner, mask_runner -FILE_PATTERN = image, weight, pipeline_flag -NUMBERING_SCHEME = -0000000-0 - -EXEC_PATH = sex -DOT_SEX_FILE = $SP_CONFIG/default_exp.sex -DOT_PARAM_FILE = $SP_CONFIG//default.param -DOT_CONV_FILE = $SP_CONFIG/default.conv - -WEIGHT_IMAGE = True -FLAG_IMAGE = True -PSF_FILE = False -DETECTION_IMAGE = False -DETECTION_WEIGHT = False - -ZP_FROM_HEADER = True -ZP_KEY = PHOTZP - -BKG_FROM_HEADER = False -CHECKIMAGE = BACKGROUND - -SUFFIX = sexcat - -MAKE_POST_PROCESS = FALSE -``` - -On success, SEXtractor catalogue FITS files are produced. - -### Select stars - -**Module:** setools -**Parent:** sextractor_runner -**Input:** sextractor catalog -**Output:** masked sextractor catalogue - -For the star selection we use the size-magnitude plane. We first find the -stellar locus, by computing the FWHM mode of all objects. Objects are selected -within some range in FWHM around the mode, and within a magnitude range. - -Apart from the standard `INPUT_MODULE`, `FILE_PATTERN`, and `NUMBERING_SCHEME` entries, a file path for selection -configuration file are given, containing the selection criteria for the star candidate sample: -```ini -[SETOOLS_RUNNER] -INPUT_MODULE = sextractor_runner - -# Note: Make sure this does not match the SExtractor background images -# (sexcat_background*) -FILE_PATTERN = sexcat_sexcat -NUMBERING_SCHEME = -0000000-0 -SETOOLS_CONFIG_PATH = $SP_CONFIG/star_selection.setools -``` -The selection config file `$SP_CONFIG/star_selection.setools` first defines a pre-selection (also "filter" or "mask"), -such that the subsequent computation of the mode is more stable: -```ini -[MASK:preselect] -MAG_AUTO > 0 -MAG_AUTO < 21 -FWHM_IMAGE > 0.3 / 0.187 -FWHM_IMAGE < 1.5 / 0.187 -FLAGS == 0 -IMAFLAGS_ISO == 0 -NO_SAVE -``` -> Note the normalisation by the pixel scale to express **FWHM_IMAGE** in arc seconds. - -The star sample is then selected in a smaller magnitude range, around the mode of the preselected sample, as follows: -```ini -[MASK:star_selection] -MAG_AUTO > 18 -MAG_AUTO < 22 -# NOTE : unit is pixel -FWHM_IMAGE <= mode(FWHM_IMAGE{preselect}) + 0.2 -FWHM_IMAGE >= mode(FWHM_IMAGE{preselect}) - 0.2 -FLAGS == 0 -IMAFLAGS_ISO == 0 -``` -From this selection, two random sub-samples are created with a number ratio of 80:20. -The first, larger sample is to [create the PSF model](#create-psf-model). The second, smaller -sample serves to [validate the PSF model](#validation-tests): -```ini -[RAND_SPLIT:star_split] -RATIO = 20 -MASK = star_selection -``` -In addition, the selection config file can contain instructions to create plots and -statistics of the selected population(s). The former can be scatter plots and histograms, -the former can include mean, mode, extrema, and standard deviation -of any quantity from the SExtractor input catalogue, the number of selected objects, etc.. - -On success, masked SExtractor catalogues are created in `mask`, plots are put in `plots`, -and text files with the computed statistics in `stats`. - -The following plots show an example, CCD #10 of exposure 2113737. +The following plots show an example of a single CCD, in the center of the focal plane. | Size-magnitude plot | Star magnitude histogram | Stars in CCD (mag) | Stars in CCD (size) | | --- | --- | --- | --- | | | | | | | The stellar locus is well-defined | Magnitude distribution looks reasonable | Stars are relatively homogeneously distributed over the CCD | The uniform and small seeing of CFHT is evident | -To contrast the last plot, here is the case of CCD #35 (lower right corner), which shows a known (but yet unexplained) lack of stars +To contrast the last plot, here is the case of the CCD in the lower right corner, which shows a known (but yet unexplained) lack of stars in the lower parts: -The statistics output file, also for CCD #10 is: +The statistics output file for the center CCD #10: ```bash (shapepipe) dap ~/ShapePipeRun $ cat output/shapepipe_run_2020-03-05_10-00-26/setools_runner/output/stat/star_stat-2113737-10.txt # Statistics @@ -408,376 +281,45 @@ to create histograms (as `.txt` tables and `.png` plots) in the directory `stats Note that `stats_global` read all `SETool` output stats files found in a given input directory tree. It can thus produce histogram combining several runs. -### Create PSF model - -**Module:** psfex -**Parent:** setools_runner -**Input:** setools_star_selection -**Output:** star catalogue, psf file - -The PSF modeling is done with `PSFEx`. The corresponding config file `$SP_CONFIG/config_exp_Psm.ini` section is noteworthy -in the specification of the input file pattern, which matches the first random sub-sample defined in the previous section. -The number has to match the previously defined ratio. - -Then, a `PSFEx` configuration path is given: -```ini -[PSFEX_RUNNER] -INPUT_MODULE = setools_runner -FILE_PATTERN = star_split_ratio_80 -NUMBERING_SCHEME = -0000000-0 -EXEC_PATH = psfex -DOT_PSFEX_FILE = $SP_CONFIG/default.psfex -``` -On success, FITS files containing the star catalalogue (`psfex_cat-*.cat`) and the PSF at -the positions of the 80% star sub-sample (`star_split_ratio_80-*.psf`) are created. - -### Validation tests - -#### Interpolate PSF to star positions - -**Module:** psfinterp_runner -**Parent**: psfex, setools -**Input:** star catalogue, psfex_catalog -**Output:** star catalogue - -The interpolation of the PSF on single exposures alone is not required for shape -measurement. But to carry validation tests on the PSF model we need the -know the PSF at the position of the stars used. For that we run the module -`psfinterp_runner`. Input files are the PSF, the positions of the 20% star sub-sample, and the PSF catalog. -The key `MODE` is set to `VALIDATION`, column names for the position parameters (SExtractor output) are -given next, and `GET_SHAPES` is True to output the PSF ellipticity using moments. And important -parameter is `STAR_THRES`, the lower limit of the number of stars per CCD to be considered to have -a valid PSF model. We set this to 22, corresponding to 20 for the 80% sub-sample used to construct the PSF. -CCDs with less stars will be discarded later on for the shape measurement. An additional cut is performed -on the chi^2 value of the PSF fit, the lower limit given by `CHI2_THRESH`. - -```ini -[PSFEXINTERP_RUNNER] -INPUT_MODULE = psfex_runner, setools_runner -FILE_PATTERN = star_split_ratio_80, star_split_ratio_20, psfex_cat -FILE_EXT = .psf, .fits, .cat -NUMBERING_SCHEME = -0000000-0 - -MODE = VALIDATION -POSITION_PARAMS = XWIN_IMAGE,YWIN_IMAGE -GET_SHAPES = True -STAR_THRESH = 22 -CHI2_THRESH = 2 -``` - -On success, validation PSF catalogues are created. - - - -#### Merge PSF catalogues -**Module:** merge_star_cat_runner -**Parent**: psfexinterp -**Input:** psfex cataloguefex -**Output:** merged SExtractor catalogue +## Galaxy selection -The star and PSF catalogues from the previous module are now merged -into a single FITS file, encoding the CCD number, with output directory given -in the module section: -```ini -OUTPUT_PATH = $SP_RUN/psf_validation -``` -The (single) output file is then `SP_RUN/psf_validation/full_starcat.fits`. (No output is produced in the actual -module output directory.) - -#### Plot focal-plane ellipticity, size and residuals - -Outside the pipeline, create plots of PSF, model, and residual -ellipticity and shape: +The focus of the next step, ```bash - ``` - -On success, `png` files with plot of the focal plane are created in `$SP_RUN/psf_validation`. - -## Process stacked images - -We leave the successfully processed single exposures for the moment, and turn our attention to the stacks. The first objective is to detect sources on the stacks, which provide a higher signal-to-noise ratio compared to detecting on the single exposures. For all further processing step, the information from the single exposures at the detected positions is used, for example the PSF model, and the object postage stamps. - -### Mask stacks - -**Module:** mask_runner -**Parent:** none -**Input:** stack image, stack weight [, star catalogue) -**Output:** stack flag - -This is the analogue of the single-exposure mask module (see [Mask images](#mask-images)) for the stacks. -The example module config file is `$SP_CONFIG/config_mask_tile.ini`, pointing to the tile-specific -mask configuration file: -```ini -[MASK_RUNNER] -MASK_CONFIG_PATH = $SP_CONFIG/config.mask +job_sp TILE_ID -j 16 ``` -The mask configuration file has a few differences compared to the single-exposure one. -First, no border region needs to be flagged. Second, no external flag files exist, -and third, the temporary directory should be different: -```ini -[BORDER_PARAMETERS] -BORDER_MAKE = False -BORDER_WIDTH = 0 -[EXTERNAL_FLAG] -EF_MAKE = False -[OTHER] -TEMP_DIRECTORY = .temp_tiles -``` - -> Note: On the cc the star catalogues created by `create_star_cat` did not -work with the pipeline. Thus for the moment, the mask package needs to be run -on the login node. - -On success, flag files containing the mask information are created. - -### Extract sources on stacks - -**Module:** sextractor_runner (in multi-epoch mode) -**Parent:** mask_runner -**Input:** tile_image, tile_weight, tile_flag -**Output:** SExtractor catalogue with multi-epoch information +is the selection of galaxies as extended objects compared to the PSF. +First, the PSF model is interpolated to galaxy positions, according to the PSF model +with `psfex_interp` or `mccd_interp`. Next, postage stamps around galaxies +of the weights maps are created via `vignetmaker`. Then, the spread model +is computed by the 'spread_model' module. Finally, postage stamps +around galaxies of single-exposure data is extracted with another call +to `vignetmaker`. -To detect a maximum of sources on the tiles, we set a low detection threshold. -In addition, a post-processing step is run to find all epochs that contributed -to a given detected object. The different entries compared to the -single-exposure case (see [Extract sources](#extract-sources)) are thus: -```ini -[SEXTRACTOR_RUNNER] -# Point to the tile-specific SExtractor parameter file -DOT_SEX_FILE = $SP_CONFIG/default_tile.sex -# Necessary for tiles, to enable multi-exposure processing -MAKE_POST_PROCESS = TRUE +## Shape measurement -# Multi-epoch mode: Path to file with single-exposure WCS header information -LOG_WCS = $SP_Run/output/log_exp_headers.sqlite - -# World coordinate keywords, SExtractor output. Format: KEY_X,KEY_Y -WORLD_POSITION = XWIN_WORLD, YWIN_WORLD - -# Number of pixels in x,y of a CCD. Format: Nx,Ny -CCD_SIZE = 33, 2080, 1, 4612 - -``` -The different entries in the SExtractor parameter file are -```ini -DETECT_THRESH 1.5 # or , in mag.arcsec-2 -ANALYSIS_THRESH 1.5 # or , in mag.arcsec-2 -DEBLEND_MINCONT 0.0005 # Minimum contrast parameter for deblending -BACK_TYPE MANUAL # AUTO or MANUAL -``` -On success, output FITS SExtractor files are created. They have the following format: -```python -HDU Name Ver Type Cards Dimensions Format - 0 PRIMARY 1 PrimaryHDU 4 () - 1 LDAC_IMHEAD 1 BinTableHDU 12 1R x 1C [8560A] - 2 LDAC_OBJECTS 1 BinTableHDU 180 25133R x 45C [1J, 1I, 1E, 1E, ...] - 3 EPOCH_0 1 BinTableHDU 16 25133R x 3C [1J, 7A, 1J] - 4 EPOCH_1 1 BinTableHDU 16 25133R x 3C [1J, 7A, 1J] - ... -``` -The first 3 HDUs correspond to the usual SExtractor output. The following HDUs -`EPOCH_` contain the single-exposure information contributing to the objects -on the stack, one HDU for each epoch. For those HDUs the columns are: -* **NUMBER**: object id atributed by SExtractor -* **EXP_NAME**: name of the single exposure (same for all objects of one HDU) -* **CCD_N**: CCD number in which the object is following the MegaCam numbering (set to -1 if the object is not - on the single exposure) - -These additionnal HDUs contain all the multi-epoch informations we need for the -following processing steps. - -On success, multi-epoch SExtractor catalogues are created. - -### Interpolate multi-epoch PSF - -**Module:** psfexinterp_runner -**Parent:** SExtractor (in multi-epoch postprocessing mode) -**Input:** SExtractor catalog with multi-epoch information -**Output:** tile PSF dictionary - -This step interpolates the PSF to the positions of all detected sources on the -tiles for all epochs on which the object is imaged. -The run mode has to be set to multi-epoch. In addition, the single-exposure -PSF information needs to be read. Unfortunately, at present, this cannot be -set in the config file as general pointer to the corresponding files from the single-exposure run. -The simplest way is to find the output directory of the star PSF files, and create a symbolic link: +The call ```bash -input_psfex=`find . -name star_selection-*.psf | head -n 1` -ln -s `dirname $input_psfex` input_psfex -``` -and to provide the link name in the config file: -```ini -[PSFEXINTERP_RUNNER] -MODE = MULTI-EPOCH -ME_DOT_PSF_DIR = input_psfex -``` - -On success, `sqlite` output data bases are created containing the PSF on -vignets (postage stamps). The structure is similar -to a dictionary with the following format: -```python -{'object_id': {'exp_name-CCD_number' : {'VIGNET': numpy.ndarray, 'SHAPES': {}}} +job_sp TILE_ID -j 32 ``` -For example: -```python -{'1': {'2104127-35': {'VIGNET': numpy.ndarray, 'SHAPES': {}}, - '2105224-13': {'VIGNET': numpy.ndarray, 'SHAPES': {}}, - '2105382-3': {'VIGNET': numpy.ndarray, 'SHAPES': {}}}, - '2': {'2104127-33': {'VIGNET': numpy.ndarray, 'SHAPES': {}}, - '2105224-12': {'VIGNET': numpy.ndarray, 'SHAPES': {}}, - '2105382-2': {'VIGNET': numpy.ndarray, 'SHAPES': {}}}, - ...} -``` - -### Create weight postage stamps - -**Module:** vignetmaker_runner -**Parent:** sextractor_runner (in multi-epoch mode) -**Input:** SExtractor catalog with multi-epoch information, tile weight -**Output:** vignet FITS table - -This modules is a pre-processing step to compute the spread model. This galaxy -classification computation, performed in the nex step, requires -* The vignets of the objects on the stack -* The PSF information at the objects' location -* The vignets of the weight image at the objects' location - -The first two outputs have already been obtained (see [Extract sources on stacks](#extract-sources-on-stacks) and []Interpolate multi-epoch PSF(#interpolate-multi-epoch-psf)), thus we only need to extract the -weights. These are needed to weigh the object images for the comparison to the weighted single-exposure PSFs -in the spread model classification. - -On success, FITS tables with vignets containing the weight for each object are created. - -### Compute spread model - -**Module:** spread_model_runner -**Parent:** psfex_runner (single-exposure), psfexinterp_runner (tile), vignetmaker_runner -**Input:** psfex catalogue, tile psf dictionary, weight vignet -**Output:** SExtractor catalogue - -The spread model for each object is computed, which serves to classify a -sub-set of detected objects on the tiles as galaxies. - -> Note: The effective PSF on the tiles is approximated by a weighted sum -of the single-exposure PSFs. The weight for each epoch is the average over -the postage stamp. - -On success, SExtractor catalogues with galaxy number, magnitude, spread model, -and an error estimate is produced. - -### Create single-exposure postage stamps - -**Module:** vignetmaker_runner2 -**Parent:** sextractor_runner -**Input:** sextractor_catalog -**Output:** single-exposure vignet dictionary - -This second iteration of the vignet creation module is the last step in -preparation for galaxy shape measurement. Multi-epoch shape measurement requires -* The SExtractor tile catalog with spread-model information -* The vignets of single-exposure images for tile-detected objects -* The vignets of single-exposure weights at the position of tile-detected objects -* The vignets of single-exposure flags at the position of tile-detected objects -* The vignets of the single-exposure background images at the position of tile- - detected objects -* The vignets of the single-exposure PSFs - -The first and last items of the list were already obtained (see [Compute spread model](#compute-spread-model) -and [Interpolate multi-epoch PSF](#interpolate-multi-epoch-psf)). The missing middle three products are -thus to be extracted here. For technical reasons, we have to use for the -moment the module `vignetmaker_runner2`, run in `MULTI-EPOCH` mode. To work -in tile coordinates, we need spherical WORLD coordinates instead of Cartesian IMAGE (pixel) -units: -```ini -[VIGNETMAKER_RUNNER2] - -MODE = MULTI-EPOCH - -# Coordinate frame type, one in PIX (pixel frame), SPHE (spherical coordinates) -COORD = SPHE - -# Coordinate frame type, one in PIX (pixel frame), SPHE (spherical coordinates) -COORD = SPHE -POSITION_PARAMS = XWIN_WORLD, YWIN_WORLD - -# Additional parameters for path and file pattern corresponding to single-exposure -# run outputs -ME_IMAGE_DIR = input_split_exp, input_split_exp, input_split_exp, input_sextractor -ME_IMAGE_PATTERN = flag, image, weight,s excat_background -``` -The last entries indicate four input paths and corresponding file patterns, for: -single_exposure single-CCD images, weights, flags (created in [Split images](#split-images)), and -single-exposure background vignet file (created in [Extract sources](#extract-sources)). - -On success, `sqlite` dictionaries with vignets for the image, weight, flag, and -background are created. - -### Multi-epoch shape measurement with `ngmix` +computes galaxy shapes using the multi-epoch model-fitting method `ngmix`. At the same time, +shapes of artifically sheared galaxies are obtained for metacalibration. -**Module:** ngmix_runner -**Parent:** -**Input:** sextractor_catalog, single_exp_image_vignet, single_exp_bkg_vignet, -tile_psf, single_exp_weight_vignet, single_exp_flag_vignet -**Output:** SExtractor catalogue -Now we are finally ready to run the shape measurement. At the moment this is done with algorithms largely based on NGMIX, and extensively using `galsim` classes. -Here is a commented example config file for the pipeline: +## Paste catalogues -```ini -[NGMIX_RUNNER] - -# Create with the split_exp_hdu module -LOG_WCS = $SP_CONFIG/output/log_exp_headers.npy -``` - -### Create final catalog - -**Module:** make_catalog_runner -**Inputs:** sextractor_catalog, spread_model, tile_psf, ngmix_catalog - -We finally merge all the results into one catalog per tiles. Here is a commented example config file for the pipeline : - -```ini -[MAKE_CATALOG_RUNNER] - -# If true will add a column in the final catalog : "SPREAD_CLASS" -SM_DO_CLASSIFICATION = True -# The classification is done by computing : -# classif = SPREAD_MODEL + 5/3 * SPREADERR_MODEL -# The cut for the star : |classif| < SM_STAR_STRESH -SM_STAR_STRESH = 0.003 -# The cut for the galaxies : classif > SM_GAL_THRESH -SM_GAL_THRESH = 0.01 +The last real processing step, +```bash +job_sp TILE_ID -j 64 ``` +which pastes previously obtained information into a 'final' catalogue via `make_cat`. +This includes galaxy detection and basic measurement parameters, the PSF model at +galaxy positions, the spread-model classification, and the shape measurement. -**Tips :** At the end we have one catalog per tiles. Here are a piece of code to merge all of them (assuming you are in the directory with all the catalogs) : - -```python -import os -import numpy as np -from shapepipe.pipeline import file_io as io - -all_file = os.listdir() - -cat = io.FITSCatalog(all_file[0]) -cat.open() - -data = np.copy(cat.get_data()) - -cat.close - -for f in all_file[1:]: - cat = io.FITSCatalog(f) - cat.open() - - data = np.concatenate((data, np.copy(cat.get_data()))) - - cat.close() - -final_catalog = io.FITSCatalog('final_cat.fits', open_mode=io.BaseCatalog.OpenMode.ReadWrite) - -final_catalog.save_as_fits(data, ext_name='RESULTS') +## Upload results +Optionally, after the pipeline is finished, results can be uploaded to VOspace via +```bash +job_sp TILE_ID -j 128 ``` From 90f2a6fccad4d14502f89bb1c6371b569998e864 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Thu, 3 Mar 2022 08:51:29 +0100 Subject: [PATCH 04/27] no change --- example/cfis/config_tile_Sx_exp_psfex.ini | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/example/cfis/config_tile_Sx_exp_psfex.ini b/example/cfis/config_tile_Sx_exp_psfex.ini index 110060f40..9db7146c0 100644 --- a/example/cfis/config_tile_Sx_exp_psfex.ini +++ b/example/cfis/config_tile_Sx_exp_psfex.ini @@ -19,8 +19,10 @@ RUN_NAME = run_sp_tile_Sx_exp_SxSePsf [EXECUTION] # Module name, single string or comma-separated list of valid module runner names -MODULE = sextractor_runner, sextractor_runner, setools_runner, - psfex_runner, psfex_interp_runner +MODULE = sextractor_runner, + sextractor_runner, setools_runner, + psfex_runner, psfex_interp_runner, + # Run mode, SMP or MPI MODE = SMP From b59c4e18ae56ce9ae1efc87922170cccc38953f9 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Thu, 3 Mar 2022 11:53:26 +0100 Subject: [PATCH 05/27] psfex update (minor bug) --- example/cfis/config_MaMa.ini | 4 ++-- example/cfis/config_tile_MiViSmVi.ini | 8 ++++---- example/cfis/config_tile_PiViSmVi.ini | 10 +++++----- example/cfis/config_tile_Sx_exp_psfex.ini | 4 +--- shapepipe/modules/psfex_runner.py | 2 +- 5 files changed, 13 insertions(+), 15 deletions(-) diff --git a/example/cfis/config_MaMa.ini b/example/cfis/config_MaMa.ini index ec407b720..102a9aa5e 100644 --- a/example/cfis/config_MaMa.ini +++ b/example/cfis/config_MaMa.ini @@ -79,7 +79,7 @@ USE_EXT_FLAG = False USE_EXT_STAR = False # File name suffix for the output flag files (optional) -SUFFIX = pipeline +PREFIX = pipeline ### Mask exposures [MASK_RUNNER_RUN_2] @@ -102,4 +102,4 @@ USE_EXT_FLAG = True USE_EXT_STAR = False # File name suffix for the output flag files (optional) -SUFFIX = pipeline +PREFIX = pipeline diff --git a/example/cfis/config_tile_MiViSmVi.ini b/example/cfis/config_tile_MiViSmVi.ini index f42c0ac61..9d2d5800e 100644 --- a/example/cfis/config_tile_MiViSmVi.ini +++ b/example/cfis/config_tile_MiViSmVi.ini @@ -117,8 +117,8 @@ POSITION_PARAMS = XWIN_IMAGE,YWIN_IMAGE # Vignet size in pixels STAMP_SIZE = 51 -# Output file name prefix, file name is _vignet.fits -SUFFIX = weight +# Output file name prefix, file name is _vignet.fits +PREFIX = weight [SPREAD_MODEL_RUNNER] @@ -171,8 +171,8 @@ POSITION_PARAMS = XWIN_WORLD,YWIN_WORLD # Vignet size in pixels STAMP_SIZE = 51 -# Output file name prefix, file name is vignet.fits -SUFFIX = +# Output file name prefix, file name is vignet.fits +PREFIX = # Additional parameters for path and file pattern corresponding to single-exposure # run outputs diff --git a/example/cfis/config_tile_PiViSmVi.ini b/example/cfis/config_tile_PiViSmVi.ini index 96946a171..2c68816fc 100644 --- a/example/cfis/config_tile_PiViSmVi.ini +++ b/example/cfis/config_tile_PiViSmVi.ini @@ -124,13 +124,13 @@ POSITION_PARAMS = XWIN_IMAGE,YWIN_IMAGE # Vignet size in pixels STAMP_SIZE = 51 -# Output file name prefix, file name is _vignet.fits -SUFFIX = weight +# Output file name prefix, file name is _vignet.fits +PREFIX = weight [SPREAD_MODEL_RUNNER] -INPUT_DIR = last:sextractor_runner_run_1, psfex_interp_runner, vignetmaker_runner +INPUT_DIR = last:sextractor_runner_run_1, last:psfex_interp_runner, last:vignetmaker_runner_run_1 FILE_PATTERN = sexcat_sexcat, galaxy_psf, weight_vignet @@ -178,8 +178,8 @@ POSITION_PARAMS = XWIN_WORLD,YWIN_WORLD # Vignet size in pixels STAMP_SIZE = 51 -# Output file name prefix, file name is vignet.fits -SUFFIX = +# Output file name prefix, file name is vignet.fits +PREFIX = # Additional parameters for path and file pattern corresponding to single-exposure # run outputs diff --git a/example/cfis/config_tile_Sx_exp_psfex.ini b/example/cfis/config_tile_Sx_exp_psfex.ini index 9db7146c0..2a4ab6d32 100644 --- a/example/cfis/config_tile_Sx_exp_psfex.ini +++ b/example/cfis/config_tile_Sx_exp_psfex.ini @@ -19,9 +19,7 @@ RUN_NAME = run_sp_tile_Sx_exp_SxSePsf [EXECUTION] # Module name, single string or comma-separated list of valid module runner names -MODULE = sextractor_runner, - sextractor_runner, setools_runner, - psfex_runner, psfex_interp_runner, +MODULE = sextractor_runner, sextractor_runner, setools_runner, psfex_runner, psfex_interp_runner # Run mode, SMP or MPI diff --git a/shapepipe/modules/psfex_runner.py b/shapepipe/modules/psfex_runner.py index 8c4ac8eea..c07985c9d 100644 --- a/shapepipe/modules/psfex_runner.py +++ b/shapepipe/modules/psfex_runner.py @@ -67,7 +67,7 @@ def psfex_runner( stderr, stdout = execute(command_line) # Parse psfex errors - stdout, stderr = psfex_call.parse_errors(stderr, stdout) + stdout, stderr = PSFExCaller.parse_errors(stderr, stdout) # Return stdout and stderr return stdout, stderr From 3ad597e9a802697525da76d436491e42e83f27a5 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Thu, 3 Mar 2022 13:41:24 +0100 Subject: [PATCH 06/27] psfex running --- example/cfis/config_make_cat_psfex.ini | 8 ++++---- example/cfis/config_tile_Ng_template.ini | 15 +++++++-------- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/example/cfis/config_make_cat_psfex.ini b/example/cfis/config_make_cat_psfex.ini index 14aaf8212..09c9cf1de 100644 --- a/example/cfis/config_make_cat_psfex.ini +++ b/example/cfis/config_make_cat_psfex.ini @@ -34,7 +34,7 @@ LOG_NAME = log_sp RUN_LOG_NAME = log_run_sp # Input directory, containing input files, single string or list of names with length matching FILE_PATTERN -INPUT_DIR = last:sextractor_runner, last:spread_model_runner, last:psfex_interp_runner, last:merge_sep_cats_runner +INPUT_DIR = . # Output directory OUTPUT_DIR = ./output @@ -54,14 +54,15 @@ TIMEOUT = 96:00:00 [MAKE_CAT_RUNNER] +# Input directory, containing input files, single string or list of names with length matching FILE_PATTERN +INPUT_DIR = last:sextractor_runner_run_1, last:spread_model_runner, last:psfex_interp_runner, last:merge_sep_cats_runner + # Input file pattern(s), list of strings with length matching number of expected input file types # Cannot contain wild cards FILE_PATTERN = sexcat_sexcat, sexcat_sm, galaxy_psf, ngmix -#, galsim # FILE_EXT (optional) list of string extensions to identify input files FILE_EXT = .fits, .fits, .sqlite, .fits -#, .fits # Numbering convention, string that exemplifies a numbering pattern. # Matches input single exposures (with 'p' removed) @@ -76,4 +77,3 @@ SM_GAL_THRESH = 0.01 TILE_LIST = $SP_CONFIG/tiles_all_order.txt SHAPE_MEASUREMENT_TYPE = ngmix -#, galsim diff --git a/example/cfis/config_tile_Ng_template.ini b/example/cfis/config_tile_Ng_template.ini index e6f56d332..f30a890b6 100644 --- a/example/cfis/config_tile_Ng_template.ini +++ b/example/cfis/config_tile_Ng_template.ini @@ -19,7 +19,6 @@ RUN_DATETIME = False # Module name, single string or comma-separated list of valid module runner names MODULE = ngmix_runner -#, galsim_shapes_v2_runner # Parallel processing mode, SMP or MPI MODE = SMP @@ -35,11 +34,7 @@ LOG_NAME = log_sp RUN_LOG_NAME = log_run_sp # Input directory, containing input files, single string or list of names -INPUT_DIR = last:get_images_runner_run_1 - -FILE_PATTERN = sexcat_sexcat, image_vignet, sexcat_background_vignet, galaxy_psf, weight_vignet, flag_vignet - -FILE_EXT = .fits, .sqlite, .sqlite, .sqlite, .sqlite, .sqlite +INPUT_DIR = . # Output directory OUTPUT_DIR = $SP_RUN/output @@ -60,11 +55,15 @@ TIMEOUT = 96:00:00 # Model-fitting shapes with ngmix [NGMIX_RUNNER] +INPUT_DIR = last:sextractor_runner_run_1,last:X_interp_runner,last:vignetmaker_runner_run_2 + +FILE_PATTERN = sexcat_sexcat, image_vignet, sexcat_background_vignet, galaxy_psf, weight_vignet, flag_vignet + +FILE_EXT = .fits, .sqlite, .sqlite, .sqlite, .sqlite, .sqlite + # NUMBERING_SCHEME (optional) string with numbering pattern for input files NUMBERING_SCHEME = -000-000 -INPUT_DIR = last:sextractor_runner_run_2,last:X_interp_runner,last:vignetmaker_runner_run_2 - # Multi-epoch mode: Path to file with single-exposure WCS header information LOG_WCS = $SP_RUN/output/log_exp_headers.sqlite From 7f3ee115cf9a21e52e9b16d999e51b68bdde9aff Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Thu, 3 Mar 2022 19:02:09 +0100 Subject: [PATCH 07/27] updated MCCD config files --- example/cfis/config_tile_MiViSmVi.ini | 8 +- example/cfis/config_tile_PiViSmVi.ini | 2 +- example/cfis/config_tile_Sx_exp_mccd.ini | 136 ++++++++++++----------- 3 files changed, 75 insertions(+), 71 deletions(-) diff --git a/example/cfis/config_tile_MiViSmVi.ini b/example/cfis/config_tile_MiViSmVi.ini index 9d2d5800e..2cd683816 100644 --- a/example/cfis/config_tile_MiViSmVi.ini +++ b/example/cfis/config_tile_MiViSmVi.ini @@ -57,7 +57,7 @@ TIMEOUT = 96:00:00 [MCCD_INTERP_RUNNER] -INPUT_DIR = last:sextractor_runner_run_2 +INPUT_DIR = last:sextractor_runner_run_1 FILE_PATTERN = sexcat_sexcat @@ -92,7 +92,7 @@ ME_LOG_WCS = $SP_RUN/output/log_exp_headers.sqlite # Create vignets for tiles weights -INPUT_DIR = last:sextractor_runner_run_2, last:uncompress_fits_runner +INPUT_DIR = last:sextractor_runner_run_1, last:uncompress_fits_runner FILE_PATTERN = sexcat_sexcat, CFIS_weight @@ -123,7 +123,7 @@ PREFIX = weight [SPREAD_MODEL_RUNNER] -INPUT_DIR = last:sextractor_runner_run_2, last:mccd_interp_runner, last:vignetmaker_runner_run_1 +INPUT_DIR = last:sextractor_runner_run_1, last:mccd_interp_runner, last:vignetmaker_runner_run_1 FILE_PATTERN = sexcat_sexcat, galaxy_psf, weight_vignet @@ -146,7 +146,7 @@ OUTPUT_MODE = new # Create multi-epoch vignets for tiles corresponding to # positions on single-exposures -INPUT_DIR = last:sextractor_runner_run_2 +INPUT_DIR = last:sextractor_runner_run_1 FILE_PATTERN = sexcat_sexcat diff --git a/example/cfis/config_tile_PiViSmVi.ini b/example/cfis/config_tile_PiViSmVi.ini index 2c68816fc..920ee61d2 100644 --- a/example/cfis/config_tile_PiViSmVi.ini +++ b/example/cfis/config_tile_PiViSmVi.ini @@ -183,6 +183,6 @@ PREFIX = # Additional parameters for path and file pattern corresponding to single-exposure # run outputs -ME_IMAGE_DIR = split_exp_runner, split_exp_runner, split_exp_runner, sextractor_runner_run_2 +ME_IMAGE_DIR = split_exp_runner, split_exp_runner, split_exp_runner, sextractor_runner_run_1 ME_IMAGE_PATTERN = flag, image, weight, sexcat_background ME_LOG_WCS = $SP_RUN/output/log_exp_headers.sqlite diff --git a/example/cfis/config_tile_Sx_exp_mccd.ini b/example/cfis/config_tile_Sx_exp_mccd.ini index 2497e7b28..eaccb693f 100644 --- a/example/cfis/config_tile_Sx_exp_mccd.ini +++ b/example/cfis/config_tile_Sx_exp_mccd.ini @@ -9,7 +9,7 @@ VERBOSE = True # Name of run (optional) default: shapepipe_run -RUN_NAME = run_sp_SxSePsfSx +RUN_NAME = run_sp_tile_Sx_exp_SxSePsf # Add date and time to RUN_NAME, optional, default: True ; RUN_DATETIME = False @@ -19,10 +19,10 @@ RUN_NAME = run_sp_SxSePsfSx [EXECUTION] # Module name, single string or comma-separated list of valid module runner names -MODULE = sextractor_runner, setools_runner, +MODULE = sextractor_runner, sextractor_runner, setools_runner, mccd_preprocessing_runner, mccd_fit_val_runner, merge_starcat_runner, mccd_plots_runner, - mccd_interp_runner, sextractor_runner + mccd_interp_runner # Run mode, SMP or MPI MODE = SMP @@ -56,8 +56,74 @@ TIMEOUT = 96:00:00 ## Module options +## Detection on tile [SEXTRACTOR_RUNNER_RUN_1] +INPUT_DIR = last:get_images_runner_run_1, last:uncompress_fits_runner, last:mask_runner_run_1 + +FILE_PATTERN = CFIS_image, CFIS_weight, pipeline_flag + +FILE_EXT = .fits, .fits, .fits + +# NUMBERING_SCHEME (optional) string with numbering pattern for input files +NUMBERING_SCHEME = -000-000 + +# SExtractor executable path +EXEC_PATH = sex + +# SExtractor configuration files +DOT_SEX_FILE = $SP_CONFIG/default_tile.sex +DOT_PARAM_FILE = $SP_CONFIG/default.param +DOT_CONV_FILE = $SP_CONFIG/default.conv + +# Use input weight image if True +WEIGHT_IMAGE = True + +# Use input flag image if True +FLAG_IMAGE = True + +# Use input PSF file if True +PSF_FILE = False + +# Use distinct image for detection (SExtractor in +# dual-image mode) if True +DETECTION_IMAGE = False + +# Distinct weight image for detection (SExtractor +# in dual-image mode) +DETECTION_WEIGHT = False + +ZP_FROM_HEADER = False + +BKG_FROM_HEADER = False + +# Type of image check (optional), default not used, can be a list of +# BACKGROUND, BACKGROUND_RMS, INIBACKGROUND, +# MINIBACK_RMS, -BACKGROUND, #FILTERED, +# OBJECTS, -OBJECTS, SEGMENTATION, APERTURES +#CHECKIMAGE = BACKGROUND + +# File name suffix for the output sextractor files (optional) +SUFFIX = sexcat + +## Post-processing + +# Necessary for tiles, to enable multi-exposure processing +MAKE_POST_PROCESS = True + +# Multi-epoch mode: Path to file with single-exposure WCS header information +LOG_WCS = $SP_RUN/output/log_exp_headers.sqlite + +# World coordinate keywords, SExtractor output. Format: KEY_X,KEY_Y +WORLD_POSITION = XWIN_WORLD,YWIN_WORLD + +# Number of pixels in x,y of a CCD. Format: Nx,Ny +CCD_SIZE = 33,2080,1,4612 + + +## Detection on single exposures +[SEXTRACTOR_RUNNER_RUN_2] + INPUT_DIR = last:split_exp_runner, last:mask_runner_run_2 # Input from two modules @@ -127,7 +193,7 @@ MAKE_POST_PROCESS = FALSE [SETOOLS_RUNNER] -INPUT_MODULE = sextractor_runner +INPUT_MODULE = sextractor_runner_run_2 # Note: Make sure this doe not match the SExtractor background images # (sexcat_background*) @@ -248,66 +314,4 @@ PSF_MODEL_SEPARATOR = - ME_LOG_WCS = $SP_RUN/output/log_exp_headers.sqlite -## Detection on tile -[SEXTRACTOR_RUNNER_RUN_2] - -INPUT_DIR = last:get_images_runner_run_1, last:uncompress_fits_runner, last:mask_runner_run_1 - -FILE_PATTERN = CFIS_image, CFIS_weight, pipeline_flag - -FILE_EXT = .fits, .fits, .fits - -# NUMBERING_SCHEME (optional) string with numbering pattern for input files -NUMBERING_SCHEME = -000-000 - -# SExtractor executable path -EXEC_PATH = sex - -# SExtractor configuration files -DOT_SEX_FILE = $SP_CONFIG/default_tile.sex -DOT_PARAM_FILE = $SP_CONFIG/default.param -DOT_CONV_FILE = $SP_CONFIG/default.conv - -# Use input weight image if True -WEIGHT_IMAGE = True - -# Use input flag image if True -FLAG_IMAGE = True - -# Use input PSF file if True -PSF_FILE = False -# Use distinct image for detection (SExtractor in -# dual-image mode) if True -DETECTION_IMAGE = False - -# Distinct weight image for detection (SExtractor -# in dual-image mode) -DETECTION_WEIGHT = False - -ZP_FROM_HEADER = False - -BKG_FROM_HEADER = False - -# Type of image check (optional), default not used, can be a list of -# BACKGROUND, BACKGROUND_RMS, INIBACKGROUND, -# MINIBACK_RMS, -BACKGROUND, #FILTERED, -# OBJECTS, -OBJECTS, SEGMENTATION, APERTURES -#CHECKIMAGE = BACKGROUND - -# File name suffix for the output sextractor files (optional) -SUFFIX = sexcat - -## Post-processing - -# Necessary for tiles, to enable multi-exposure processing -MAKE_POST_PROCESS = True - -# Multi-epoch mode: Path to file with single-exposure WCS header information -LOG_WCS = $SP_RUN/output/log_exp_headers.sqlite - -# World coordinate keywords, SExtractor output. Format: KEY_X,KEY_Y -WORLD_POSITION = XWIN_WORLD,YWIN_WORLD - -# Number of pixels in x,y of a CCD. Format: Nx,Ny -CCD_SIZE = 33,2080,1,4612 From d5d6f940dfa18e768ea276fe905c4295717ae632 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Fri, 4 Mar 2022 10:20:30 +0100 Subject: [PATCH 08/27] comment in config file --- example/cfis/config_tile_Sx_exp_psfex.ini | 2 +- example/cfis/default.psfex | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/example/cfis/config_tile_Sx_exp_psfex.ini b/example/cfis/config_tile_Sx_exp_psfex.ini index 9db7146c0..fc06cf023 100644 --- a/example/cfis/config_tile_Sx_exp_psfex.ini +++ b/example/cfis/config_tile_Sx_exp_psfex.ini @@ -215,7 +215,7 @@ FILE_PATTERN = star_split_ratio_80 NUMBERING_SCHEME = -0000000-0 -# Path to executable for the PSF model +# Path to executable for the PSF model (optional) EXEC_PATH = psfex # Default psfex configuration file diff --git a/example/cfis/default.psfex b/example/cfis/default.psfex index 6bc0fdc28..15af2dc29 100644 --- a/example/cfis/default.psfex +++ b/example/cfis/default.psfex @@ -66,8 +66,7 @@ CHECKPLOT_NAME fwhm, ellipticity, counts, countfrac, chi2, resi #------------------------------ Check-Images --------------------------------- -####### -## AG : This parameter is set in pipeline config file. +# Note: Check-image types can be set the ShapePipe config file, psfex_runner section ####### #CHECKIMAGE_TYPE NONE # CHI,PROTOTYPES,SAMPLES,RESIDUALS,SNAPSHOTS # or MOFFAT,-MOFFAT,-SYMMETRICAL From 37213dd6f50e34ae6ecc538c7b224584178180cb Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Fri, 4 Mar 2022 11:59:48 +0100 Subject: [PATCH 09/27] config file fix for MCCD --- example/cfis/config_tile_MiViSmVi.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/example/cfis/config_tile_MiViSmVi.ini b/example/cfis/config_tile_MiViSmVi.ini index 2cd683816..6d4ff5c4d 100644 --- a/example/cfis/config_tile_MiViSmVi.ini +++ b/example/cfis/config_tile_MiViSmVi.ini @@ -176,6 +176,6 @@ PREFIX = # Additional parameters for path and file pattern corresponding to single-exposure # run outputs -ME_IMAGE_DIR = split_exp_runner, split_exp_runner, split_exp_runner, sextractor_runner_run_1 +ME_IMAGE_DIR = split_exp_runner, split_exp_runner, split_exp_runner, sextractor_runner_run_2 ME_IMAGE_PATTERN = flag, image, weight, sexcat_background ME_LOG_WCS = $SP_RUN/output/log_exp_headers.sqlite From 64eecd3c68bbf7b3ea8848bf04d60aa54b5ab7b5 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Fri, 4 Mar 2022 12:03:44 +0100 Subject: [PATCH 10/27] config file fix for MCCD --- example/cfis/config_make_cat_mccd.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/example/cfis/config_make_cat_mccd.ini b/example/cfis/config_make_cat_mccd.ini index c8173eabd..2cb12b870 100644 --- a/example/cfis/config_make_cat_mccd.ini +++ b/example/cfis/config_make_cat_mccd.ini @@ -34,7 +34,7 @@ LOG_NAME = log_sp RUN_LOG_NAME = log_run_sp # Input directory, containing input files, single string or list of names with length matching FILE_PATTERN -INPUT_DIR = last:sextractor_runner_run_2, last:spread_model_runner, last:mccd_interp_runner, last:merge_sep_cats_runner +INPUT_DIR = last:sextractor_runner_run_1, last:spread_model_runner, last:mccd_interp_runner, last:merge_sep_cats_runner # Output directory OUTPUT_DIR = ./output From 725891062d6c3165c3e8b51889939ea29c170f61 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Fri, 4 Mar 2022 12:12:49 +0100 Subject: [PATCH 11/27] removed unused option in usage str --- scripts/sh/job_sp.bash | 1 - 1 file changed, 1 deletion(-) diff --git a/scripts/sh/job_sp.bash b/scripts/sh/job_sp.bash index 8679403c2..6cbb687e7 100755 --- a/scripts/sh/job_sp.bash +++ b/scripts/sh/job_sp.bash @@ -35,7 +35,6 @@ nsh_jobs=8 usage="Usage: $(basename "$0") [OPTIONS] TILE_ID_1 [TILE_ID_2 [...]] \n\nOptions:\n -h\tthis message\n - -e\tset environment and exit (run as '. $(basename "$0")'\n -j, --job JOB\tRunning JOB, bit-coded\n \t 1: retrieve images (online if method=vos)\n \t 2: prepare images (offline)\n From c7a43cb75c8c539497c49b42d8b884e14d4342d4 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Sat, 5 Mar 2022 18:57:22 +0100 Subject: [PATCH 12/27] Update docs/source/pipeline_tutorial.md Co-authored-by: Samuel Farrens --- docs/source/pipeline_tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index 082f4e1c0..defe70362 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -292,7 +292,7 @@ is the selection of galaxies as extended objects compared to the PSF. First, the PSF model is interpolated to galaxy positions, according to the PSF model with `psfex_interp` or `mccd_interp`. Next, postage stamps around galaxies of the weights maps are created via `vignetmaker`. Then, the spread model -is computed by the 'spread_model' module. Finally, postage stamps +is computed by the `spread_model` module. Finally, postage stamps around galaxies of single-exposure data is extracted with another call to `vignetmaker`. From 3fcb9584112e28489e5cfbf48f9c0e1c299acf23 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Sat, 5 Mar 2022 19:06:26 +0100 Subject: [PATCH 13/27] improved tutorial according to PR review --- docs/source/pipeline_tutorial.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index defe70362..a1a8466db 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -83,6 +83,11 @@ Naming and numbering of the input files can closely follow the original image na `PSFEx` and `SExtractor` produce FITS files with file exentions other than `.fits`: `.psf` for files containing PSF model information for a single CCD, and `.cat` for a PSF catalogue. +- _final_ shape catalogue + The end product of `ShapePipe` is a _final_ catalogue containing a large number of information for each galaxy, including its + shape parameters, the ellipticity components :math:`e_1` and :math:`e_2`. This catalogue also contains shapes of artificially + sheared images. This information is used in post-processing to compute calibrated shear estimates via metacalibration. + - Summary statistic files The `SETools` module that creates samples of objects according to some user-defined selection criteria (see [Select stars](#select-stars)) also outputs ASCII files with user-defined summary statistics for each CCD, for example the number of selected stars, or mean and standard deviation of their FWHM. @@ -109,7 +114,7 @@ Naming and numbering of the input files can closely follow the original image na `ShapePipe` splits the processing of CFIS images into several parts: These are the retrieval and preparation of input images, processing of single exposures, -processing of tile images, creation and upload (optional) of final shear catalogues. +processing of tile images, creation and upload (optional) of _final_ shape catalogues. The following flowchart visualised the processing parts and steps. @@ -313,7 +318,7 @@ The last real processing step, ```bash job_sp TILE_ID -j 64 ``` -which pastes previously obtained information into a 'final' catalogue via `make_cat`. +which pastes previously obtained information into a _final_ shape catalogue via `make_cat`. This includes galaxy detection and basic measurement parameters, the PSF model at galaxy positions, the spread-model classification, and the shape measurement. From 7321f0e8d71cc659a02747c12ac0f98bfe5bc90f Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Sat, 5 Mar 2022 19:06:57 +0100 Subject: [PATCH 14/27] Update docs/source/pipeline_tutorial.md Co-authored-by: Samuel Farrens --- docs/source/pipeline_tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index a1a8466db..b3403afb2 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -184,7 +184,7 @@ The command job_sp TILE_ID -j 1 ``` retrieves the image and weight corresponding to TILE_ID using the module `get_images`. -It then identifies the exposures that were used to create the tile image via the 'find_exposures` runner. +It then identifies the exposures that were used to create the tile image via the `find_exposures` runner. Finally, another call to `get_images` retrieves the exposure images, weights, and flag files. For the retrieval method the user can choose betwen From 4b14168b4c4da82238715919b3b5dfc28c5453a9 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Thu, 10 Mar 2022 13:57:30 +0100 Subject: [PATCH 15/27] tutorial clean-up --- docs/source/pipeline_tutorial.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index b3403afb2..7ca5886bb 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -83,7 +83,7 @@ Naming and numbering of the input files can closely follow the original image na `PSFEx` and `SExtractor` produce FITS files with file exentions other than `.fits`: `.psf` for files containing PSF model information for a single CCD, and `.cat` for a PSF catalogue. -- _final_ shape catalogue +- _final_ shape catalogue The end product of `ShapePipe` is a _final_ catalogue containing a large number of information for each galaxy, including its shape parameters, the ellipticity components :math:`e_1` and :math:`e_2`. This catalogue also contains shapes of artificially sheared images. This information is used in post-processing to compute calibrated shear estimates via metacalibration. @@ -93,20 +93,20 @@ Naming and numbering of the input files can closely follow the original image na files with user-defined summary statistics for each CCD, for example the number of selected stars, or mean and standard deviation of their FWHM. Example: `star_stat-2366993-18.txt` -- Tile ID list +- Tile ID list ASCII file with a tile number on each line. Used for the `get_image_runner` module to download CFIS images (see [Download tiles](#download-tiles)). - - Single-exposure name list - ASCII file with a single-exposure name on each line. Produced by the `find_exposure_runner` module to identify single exposures that were used to create - a given tile. See [Find exposures](#find-exposures)). +- Single-exposure name list + ASCII file with a single-exposure name on each line. Produced by the `find_exposure_runner` module to identify single exposures that were used to create + a given tile. See [Find exposures](#find-exposures)). -- Plots +- Plots The `SETools` module can also produce plots of the objects properties that were selected for a given CCD. The type of plot (histogram, scatter plot, ...) and quantities to plot as well as plot decorations can be specified in the selection criteria config file (see [Select stars](#select-stars)). Example: `hist_mag_stars-2104133-5.png` -- Log files +- Log files The pipeline core and all called modules write ASCII log files to disk. Examples: `process-2366993-6.log`, `log_sp_exp.log`. From 4cb6b56c6de3a39f208b5f7b293f63cc3bd2c708 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Mon, 14 Mar 2022 18:24:38 +0100 Subject: [PATCH 16/27] post_proc cleaned up; tutorial added output dir names --- docs/source/pipeline_tutorial.md | 27 +++++++++++++++++++++++++++ docs/source/post_processing.md | 21 ++++++++++++++------- 2 files changed, 41 insertions(+), 7 deletions(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index c71e25d61..295050086 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -193,6 +193,9 @@ For the retrieval method the user can choose betwen Note that internet access is required for this step if the download method is `vos`. +An output directory `run_sp_GitFeGie` is created containing the results of `get_images` for tiles (`Git`), +`find_exposures` (`Fe`), and `get_images` for exposures (`Gie`). + ## Prepare input images With @@ -203,6 +206,9 @@ the compressed tile weight image is uncompressed via the `uncompress_fits` modul (one FITS file per CCD) with `split_exp`. Finally, the headers of all single-exposure single-CCD files are merged into a single `sqlite` file, to store the WCS information of the input exposures. +Two output directories are created, `run_sp_Uz` for `uncompress_fits`, and `run_sp_exp_SpMh` for the output of the modules +`split_exp` (`Sp`) and `merge_headers` (`Mh`). + ## Mask images Run @@ -213,6 +219,8 @@ to mask tile and single-exposure single-CCD images. Both tasks are performed by Note that internet access is required for this step, since a reference star catalogue is downloaded. +The output of both masking runs are stored in the output directory `run_sp_MaMa`. + **Diagnostics:** Open a single-exposure single-CCD image and the corresponding pipeline flag in `ds9`, and display both frames next to each other. Example ```bash @@ -243,6 +251,10 @@ Next, the following tasks are run on the single-exposure single-CCD images: via a call to `psfex_interp`. For MCCD, the modules `merge_starcat`, `mccd_plots`, and `mccd_interp` are called. +The output directory for both the `mccd` and `psfex` options is `run_sp_tile_Sx_exp_SxSePsf`. +This stores the output of SExtractor on the tiles (`tile_Sx`), on the exposures (`exp_Sx`), +`setools` (`Se`), and the Psf model (`Psf`). + The following plots show an example of a single CCD, in the center of the focal plane. | Size-magnitude plot | Star magnitude histogram | Stars in CCD (mag) | Stars in CCD (size) | @@ -301,6 +313,13 @@ is computed by the `spread_model` module. Finally, postage stamps around galaxies of single-exposure data is extracted with another call to `vignetmaker`. +The output directory is +- `run_sp_MiViSmVi` if the PSF model is `mccd`; +- `run_sp_tile_PsViSmVi` for the `PSFEx` PSF model. + +This corresponds to the MCCD/PSFex interpolation (`Mi`/`Pi`), `vignetmaker` (`Vi`), `spread_model` (`Sm`), and the +second call to `vignetmaker` (`Vi`). + ## Shape measurement @@ -311,6 +330,10 @@ job_sp TILE_ID -j 32 computes galaxy shapes using the multi-epoch model-fitting method `ngmix`. At the same time, shapes of artifically sheared galaxies are obtained for metacalibration. +Shape measurement is performed in parallel for each tile, the number of processes can be specified +by the user with the option `--nsh_jobs NJOB`. This creates `NJOB` output directories `run_sp_tile_ngmix_Ngu`. +with `X` = 1 ... `NJOB` containing the result of `ngmix`. + ## Paste catalogues @@ -322,9 +345,13 @@ which pastes previously obtained information into a _final_ shape catalogue via This includes galaxy detection and basic measurement parameters, the PSF model at galaxy positions, the spread-model classification, and the shape measurement. +The output directory for this task is `run_sp_Ms` for the `merge_sep` run. + ## Upload results Optionally, after the pipeline is finished, results can be uploaded to VOspace via ```bash job_sp TILE_ID -j 128 ``` + +Both for `MCCD` and `PSFex` the output directory containing the result of `make_cat` is `run_sp_Mc`. diff --git a/docs/source/post_processing.md b/docs/source/post_processing.md index 25dee7180..5d1b04534 100644 --- a/docs/source/post_processing.md +++ b/docs/source/post_processing.md @@ -1,9 +1,16 @@ # Post-processing -This page shows all required steps of post-processing a `ShapePipe` run. Some details pertain specifically to -runs carried out on [canfar](./canfar.md), but most are general. +This page shows all required steps of post-processing the results from one or +more `ShapePipe` runs. Post-processing combines various individual 'ShapePipe' +output files, and creates joint results, for example combining individual tile +catalogues in a large sky area. The output of post-processing is a joint _shape +catalogue_, containing all required information to create a calibrated shear +catalogue via 'metacalibration'), a joint star catalogue, and PSF diagnostic plots. -1. Retrieve files with `ShapePipe` results +Some of the following steps pertain specifically to runs carried out on [canfar](./canfar.md), +but most are general. + +1. Retrieve `ShapePipe` result files For a local run on the same machine as for post-processing, nothing needs to be done. In some cases, the run was carried out on a remote machine or cluster, and the resulting `ShapePipe` @@ -59,7 +66,7 @@ At this step all required `ShapePipe` resulting output files are available in `. the previous step. For example, to create the subdir `tiles_W3` with links to result files to `all` for those tiles contained in the list `tiles_W3.txt`, do: ```bash - $SP_ROOT/scripts/python/create_sample_results.py --input_IDs tiles_W3.txt -i . all -o tiles_W3 -v + create_sample_results --input_IDs tiles_W3.txt -i . all -o tiles_W3 -v ``` The following steps will then be done in the directory `tiles_W3`. @@ -69,8 +76,8 @@ At this step all required `ShapePipe` resulting output files are available in `. ```bash post_proc_sp -p PSF ``` - to automatically perform a number of post-processing steps. Chose the PSF model with the option `-p psfex|mccd`. In detail, these are (and can also be - done individually + to automatically perform a number of post-processing steps. Chose the PSF model with the option + `-p psfex|mccd`. In detail, these are (and can also be done individually by hand): A. Analyse psf validation files @@ -93,7 +100,7 @@ At this step all required `ShapePipe` resulting output files are available in `. C. Merge final output files - Create a single main catalog: + Create a single main shape catalog: ```bash merge_final_cat -i -p -v ``` From ec964ba443ad39d1586c896133eab7d094e111db Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Tue, 15 Mar 2022 12:14:59 +0100 Subject: [PATCH 17/27] changes to tutorial, post-proc and mccd plot doc --- docs/source/pipeline_tutorial.md | 32 +++++++----- docs/source/post_processing.md | 21 +++++--- docs/source/refs.bib | 59 ++++++++++++++++++++++ shapepipe/modules/mccd_package/__init__.py | 17 +++++-- 4 files changed, 104 insertions(+), 25 deletions(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index a04c38ff9..70cccc344 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -125,7 +125,7 @@ job_sp -h ``` for all options. -This script created the subdirectory `$SP_RUN/output` to store all pipeline outputs +This script creates the subdirectory `$SP_RUN/output` to store all pipeline outputs (log files, diagnostics, statistics, output images, catalogues, single-exposure headers with WCS information). Optionally, the subdir `output_star_cat` is created by the used to store the external star catalogues for masking. This is only necessary if the pipeline is run on a cluster without internet connection to access star catalogues. In that case, the star catalogues need to be retrieved outside the pipeline, for example on a login node, and copied to `output_star_cat`. @@ -173,7 +173,7 @@ For the retrieval method the user can choose betwen Note that internet access is required for this step if the download method is `vos`. -An output directory `run_sp_GitFeGie` is created containing the results of `get_images` for tiles (`Git`), +An output directory `run_sp_GitFeGie` (in `output`) is created containing the results of `get_images` for tiles (`Git`), `find_exposures` (`Fe`), and `get_images` for exposures (`Gie`). ## Prepare input images @@ -199,16 +199,17 @@ to mask tile and single-exposure single-CCD images. Both tasks are performed by Note that internet access is required for this step, since a reference star catalogue is downloaded. -The output of both masking runs are stored in the output directory `run_sp_MaMa`. +The output of both masking runs are stored in the output directory `run_sp_MaMa`, with run 1 (2) of +`mask` corresponding to tiles (exposures). **Diagnostics:** Open a single-exposure single-CCD image and the corresponding pipeline flag in `ds9`, and display both frames next to each other. Example ```bash -ds9 output/shapepipe_run_2020-03-03_15-31-00/split_exp_runner/output/image-2113737-10.fits output/shapepipe_run_2020-03-03_17-29-34/mask_runner/output/pipeline_flag-2113737-10.fits +ds9 image-2113737-10.fits pipeline_flag-2113737-10.fits ``` Choose `zoom fit` for both frames, click `scale zscale` for the image, and `color aips0` for the flag, to display something like this: - + By eye the correspondence between the different flag types and the image can be seen. Note that the two frames might not match perfectly, since (a) WCS @@ -239,17 +240,17 @@ The following plots show an example of a single CCD, in the center of the focal | Size-magnitude plot | Star magnitude histogram | Stars in CCD (mag) | Stars in CCD (size) | | --- | --- | --- | --- | -| | | | | +| | | | | | The stellar locus is well-defined | Magnitude distribution looks reasonable | Stars are relatively homogeneously distributed over the CCD | The uniform and small seeing of CFHT is evident | To contrast the last plot, here is the case of the CCD in the lower right corner, which shows a known (but yet unexplained) lack of stars in the lower parts: - + The statistics output file for the center CCD #10: ```bash -(shapepipe) dap ~/ShapePipeRun $ cat output/shapepipe_run_2020-03-05_10-00-26/setools_runner/output/stat/star_stat-2113737-10.txt +(shapepipe) dap ~/ShapePipeRun $ cat star_stat-2113737-10.txt # Statistics Nb objects full cat = 1267 Nb stars = 160 @@ -272,7 +273,7 @@ to create histograms (as `.txt` tables and `.png` plots) in the directory `stats | Non-masked objects per CCD | Stars per CCD | FWHM mode | | --- | --- | --- | -| | | | +| | | | | No CCD with a very large masked area | No CCD with insufficient stars | Rather broad seeing distribution | Note that `stats_global` read all `SETool` output stats files found in a given input directory tree. It can thus produce histogram combining @@ -317,15 +318,19 @@ with `X` = 1 ... `NJOB` containing the result of `ngmix`. ## Paste catalogues -The last real processing step, +The last real processing step is ```bash job_sp TILE_ID -j 64 ``` -which pastes previously obtained information into a _final_ shape catalogue via `make_cat`. -This includes galaxy detection and basic measurement parameters, the PSF model at +This task first merges the `NJOB` parallel `ngmix` output files from the previous step into +one output file. Then, previously obtained information are pasted into a _final_ shape catalogue via `make_cat`. +Included are galaxy detection and basic measurement parameters, the PSF model at galaxy positions, the spread-model classification, and the shape measurement. -The output directory for this task is `run_sp_Ms` for the `merge_sep` run. +Two output directories are created. +The first one is `run_sp_Ms` for the `merge_sep` run. +The second is `run_sp_Mc` for the `make_cat` task; the name is the same for both the `MCCD` and `PSFEx` PSF model. + ## Upload results @@ -334,4 +339,3 @@ Optionally, after the pipeline is finished, results can be uploaded to VOspace v job_sp TILE_ID -j 128 ``` -Both for `MCCD` and `PSFex` the output directory containing the result of `make_cat` is `run_sp_Mc`. diff --git a/docs/source/post_processing.md b/docs/source/post_processing.md index 5d1b04534..d99fd8e50 100644 --- a/docs/source/post_processing.md +++ b/docs/source/post_processing.md @@ -57,7 +57,7 @@ but most are general. ``` On success, `ShapePipe` output `fits` and `log` files will be now in various subdirs of the `output` directory. -At this step all required `ShapePipe` resulting output files are available in `.`. +At this step all required `ShapePipe` resulting output files are available in the current working directory. 2. Optional: Split output in sub-samples @@ -83,13 +83,20 @@ At this step all required `ShapePipe` resulting output files are available in `. A. Analyse psf validation files ```bash - psf_residuals + psf_residuals -p PSF ``` with options as for `post_proc_sp`. This script identifies all psf validation files (from all processed tiles downloaded to `pwd`), creates symbolic links, merges the catalogues, and creates plots of PSF ellipticity, size, and residuals over the focal plane. - B. Prepare output directory + B. Create plots of the PSF and their residuals in the focal plane, as a diagnostic of the overall PSF model. + As a scale-dependend test, which propagates directly to the shear correlation function, the rho statistics are computed, + see {cite:p}`rowe:10` and {cite:p}`jarvis:16`, + ```bash + shapepipe_run -c /path/to/shapepipe/example/cfis/config_MsPl_PSF.ini + ``` + + C. Prepare output directory Create links to all 'final_cat' result files with ```bash @@ -98,11 +105,13 @@ At this step all required `ShapePipe` resulting output files are available in `. The corresponding output directory that is created is `output/run_sp_combined/make_catalog_runner/output`. On success, it contains links to all `final_cat` output catalogues - C. Merge final output files + D. Merge final output files Create a single main shape catalog: ```bash merge_final_cat -i -p -v ``` - Chose as input directory `input_dir` the output of step B. A default parameter file is `/path/to/shapepipe/example/cfis/final_cat.param`. - On success, the file `./final_cat.npy` is created. Depending on the number of input tiles, this file can be several tens of Gb large. + Choose as input directory `input_dir` the output of step C. A default + parameter file is `/path/to/shapepipe/example/cfis/final_cat.param`. + On success, the file `./final_cat.npy` is created. Depending on the number of + input tiles, this file can be several tens of Gb large. diff --git a/docs/source/refs.bib b/docs/source/refs.bib index 5adbc83f0..f2f5c933d 100644 --- a/docs/source/refs.bib +++ b/docs/source/refs.bib @@ -437,6 +437,47 @@ @ARTICLE{jarvis:04 adsnote = {Provided by the SAO/NASA Astrophysics Data System} } +@ARTICLE{jarvis:16, + author = {{Jarvis}, M. and {Sheldon}, E. and {Zuntz}, J. and {Kacprzak}, T. and + {Bridle}, S.~L. and {Amara}, A. and {Armstrong}, R. and {Becker}, M.~R. and + {Bernstein}, G.~M. and {Bonnett}, C. and {Chang}, C. and {Das}, R. and + {Dietrich}, J.~P. and {Drlica-Wagner}, A. and {Eifler}, T.~F. and + {Gangkofner}, C. and {Gruen}, D. and {Hirsch}, M. and {Huff}, E.~M. and + {Jain}, B. and {Kent}, S. and {Kirk}, D. and {MacCrann}, N. and + {Melchior}, P. and {Plazas}, A.~A. and {Refregier}, A. and {Rowe}, B. and + {Rykoff}, E.~S. and {Samuroff}, S. and {S{\'a}nchez}, C. and + {Suchyta}, E. and {Troxel}, M.~A. and {Vikram}, V. and {Abbott}, T. and + {Abdalla}, F.~B. and {Allam}, S. and {Annis}, J. and {Benoit-L{\'e}vy}, A. and + {Bertin}, E. and {Brooks}, D. and {Buckley-Geer}, E. and {Burke}, D.~L. and + {Capozzi}, D. and {Carnero Rosell}, A. and {Carrasco Kind}, M. and + {Carretero}, J. and {Castander}, F.~J. and {Clampitt}, J. and + {Crocce}, M. and {Cunha}, C.~E. and {D'Andrea}, C.~B. and {da Costa}, L.~N. and + {DePoy}, D.~L. and {Desai}, S. and {Diehl}, H.~T. and {Doel}, P. and + {Fausti Neto}, A. and {Flaugher}, B. and {Fosalba}, P. and {Frieman}, J. and + {Gaztanaga}, E. and {Gerdes}, D.~W. and {Gruendl}, R.~A. and + {Gutierrez}, G. and {Honscheid}, K. and {James}, D.~J. and {Kuehn}, K. and + {Kuropatkin}, N. and {Lahav}, O. and {Li}, T.~S. and {Lima}, M. and + {March}, M. and {Martini}, P. and {Miquel}, R. and {Mohr}, J.~J. and + {Neilsen}, E. and {Nord}, B. and {Ogando}, R. and {Reil}, K. and + {Romer}, A.~K. and {Roodman}, A. and {Sako}, M. and {Sanchez}, E. and + {Scarpine}, V. and {Schubnell}, M. and {Sevilla-Noarbe}, I. and + {Smith}, R.~C. and {Soares-Santos}, M. and {Sobreira}, F. and + {Swanson}, M.~E.~C. and {Tarle}, G. and {Thaler}, J. and {Thomas}, D. and + {Walker}, A.~R. and {Wechsler}, R.~H.}, + title = "{The DES Science Verification weak lensing shear catalogues}", + journal = {\mnras}, +archivePrefix = "arXiv", + eprint = {1507.05603}, + primaryClass = "astro-ph.IM", + year = 2016, + month = aug, + volume = 460, + pages = {2245-2281}, + doi = {10.1093/mnras/stw990}, + adsurl = {http://cdsads.u-strasbg.fr/abs/2016MNRAS.460.2245J}, + adsnote = {Provided by the SAO/NASA Astrophysics Data System} +} + @ARTICLE{jensen:21, author = {{Jensen}, Jaclyn and {Thomas}, Guillaume and {McConnachie}, Alan W. and {Starkenburg}, Else and {Malhan}, Khyati and {Navarro}, Julio and {Martin}, Nicolas and {Famaey}, Benoit and {Ibata}, Rodrigo and {Chapman}, Scott and {Cuillandre}, Jean-Charles and {Gwyn}, Stephen}, title = "{Uncovering fossils of the distant Milky Way with UNIONS: NGC 5466 and its stellar stream}", @@ -595,6 +636,24 @@ @ARTICLE{roberts:22 adsnote = {Provided by the SAO/NASA Astrophysics Data System} } +@ARTICLE{rowe:10, + author = {{Rowe}, B.}, + title = "{Improving PSF modelling for weak gravitational lensing using new methods in model selection}", + journal = {\mnras}, +archivePrefix = "arXiv", + eprint = {0904.3056}, + primaryClass = "astro-ph.CO", + keywords = {gravitational lensing, methods: data analysis, methods: statistical, cosmology: observations, large-scale structure of Universe}, + year = 2010, + OPTmonth = may, + volume = 404, + pages = {350-366}, + doi = {10.1111/j.1365-2966.2010.16277.x}, + adsurl = {http://adsabs.harvard.edu/abs/2010MNRAS.404..350R}, + adsnote = {Provided by the SAO/NASA Astrophysics Data System} +} + + @ARTICLE{rowe:15, author = {{Rowe}, B.~T.~P. and {Jarvis}, M. and {Mandelbaum}, R. and {Bernstein}, G.~M. and {Bosch}, J. and {Simet}, M. and {Meyers}, J.~E. and {Kacprzak}, T. and {Nakajima}, R. and {Zuntz}, J. and {Miyatake}, H. and {Dietrich}, J.~P. and {Armstrong}, R. and {Melchior}, P. and {Gill}, M.~S.~S.}, title = "{GALSIM: The modular galaxy image simulation toolkit}", diff --git a/shapepipe/modules/mccd_package/__init__.py b/shapepipe/modules/mccd_package/__init__.py index a748f49f0..408e74c33 100644 --- a/shapepipe/modules/mccd_package/__init__.py +++ b/shapepipe/modules/mccd_package/__init__.py @@ -143,11 +143,18 @@ :Output: Several plots -This module generates different plots of the input merged star catalogue. -The plots include mean shapes, where the ellipticity error is plotted as a -function of the focal plane positions. The errors are binned through all the -merged catalogue in the grid defined by ``X_GRID`` and ``Y_GRID``. The other -plots are a histogram of the ellipticity errors and the rho stats plot. +This module generates different series of plots of the input merged star catalogue. + +One series of plots are the "mean shapes", which is the PSF ellipticity and size +and their residuals as a function of the focal plane positions. Values and residuals +are combined from all merged catalogues, and binned in the focal plane according to +``X_GRID`` and ``Y_GRID``. + +Another series of plots are a histogram of the ellipticity errors. + +A third series are the rho statistics plot, which are correlation functions of +various combinations of PSF ellipticity, size, and their residuals, +see :cite:`rowe:10` and :cite:`jarvis:16`. Module-specific config file entries ----------------------------------- From 5affd2211829c80fe1db6f960ca311421d60e176 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Wed, 16 Mar 2022 17:05:47 +0100 Subject: [PATCH 18/27] get_images error msg testing --- shapepipe/modules/get_images_package/get_images.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/shapepipe/modules/get_images_package/get_images.py b/shapepipe/modules/get_images_package/get_images.py index 0cd748d60..52eae63b0 100644 --- a/shapepipe/modules/get_images_package/get_images.py +++ b/shapepipe/modules/get_images_package/get_images.py @@ -310,7 +310,7 @@ def retrieve_one(self, in_path, out_path): log_cmd = ' '.join(sys.argv) vcp = vosHandler('vcp') - print(log_cmd) + self._w_log.info(log_cmd) attempt = 0 while attempt < self._n_try: From 91c219edb4e643ee9a09741b5edd088fe65e5704 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Fri, 18 Mar 2022 19:38:13 +0100 Subject: [PATCH 19/27] Update docs/source/pipeline_tutorial.md Co-authored-by: Samuel Farrens --- docs/source/pipeline_tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index 70cccc344..4a029b8b5 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -209,7 +209,7 @@ ds9 image-2113737-10.fits pipeline_flag-2113737-10.fits ``` Choose `zoom fit` for both frames, click `scale zscale` for the image, and `color aips0` for the flag, to display something like this: - + By eye the correspondence between the different flag types and the image can be seen. Note that the two frames might not match perfectly, since (a) WCS From 591125d6bb61cd3058a00c8fd40c48ad8d09e66f Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Fri, 18 Mar 2022 19:38:34 +0100 Subject: [PATCH 20/27] Update docs/source/pipeline_tutorial.md Co-authored-by: Samuel Farrens --- docs/source/pipeline_tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index 4a029b8b5..79474762d 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -240,7 +240,7 @@ The following plots show an example of a single CCD, in the center of the focal | Size-magnitude plot | Star magnitude histogram | Stars in CCD (mag) | Stars in CCD (size) | | --- | --- | --- | --- | -| | | | | +| | | | | | The stellar locus is well-defined | Magnitude distribution looks reasonable | Stars are relatively homogeneously distributed over the CCD | The uniform and small seeing of CFHT is evident | To contrast the last plot, here is the case of the CCD in the lower right corner, which shows a known (but yet unexplained) lack of stars From d02d62423038c7491c4f23b3afde52cf6e526f14 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Fri, 18 Mar 2022 19:38:42 +0100 Subject: [PATCH 21/27] Update docs/source/pipeline_tutorial.md Co-authored-by: Samuel Farrens --- docs/source/pipeline_tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index 79474762d..6cf629f41 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -273,7 +273,7 @@ to create histograms (as `.txt` tables and `.png` plots) in the directory `stats | Non-masked objects per CCD | Stars per CCD | FWHM mode | | --- | --- | --- | -| | | | +| | | | | No CCD with a very large masked area | No CCD with insufficient stars | Rather broad seeing distribution | Note that `stats_global` read all `SETool` output stats files found in a given input directory tree. It can thus produce histogram combining From e3f36e3c18abfd541fde206acb8a1972b7deb312 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Fri, 18 Mar 2022 19:44:45 +0100 Subject: [PATCH 22/27] Update docs/source/pipeline_tutorial.md Co-authored-by: Samuel Farrens --- docs/source/pipeline_tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index 6cf629f41..2c21ac3ea 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -246,7 +246,7 @@ The following plots show an example of a single CCD, in the center of the focal To contrast the last plot, here is the case of the CCD in the lower right corner, which shows a known (but yet unexplained) lack of stars in the lower parts: - + The statistics output file for the center CCD #10: ```bash From 5b6b8546b29b06f9c78ec6be62640321e9c70762 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Wed, 23 Mar 2022 09:18:44 +0100 Subject: [PATCH 23/27] removed prompt from bash example --- docs/source/pipeline_tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index 70cccc344..b5814fe5c 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -250,7 +250,7 @@ in the lower parts: The statistics output file for the center CCD #10: ```bash -(shapepipe) dap ~/ShapePipeRun $ cat star_stat-2113737-10.txt +cat star_stat-2113737-10.txt # Statistics Nb objects full cat = 1267 Nb stars = 160 From c8100a2d9cb925105c732a297cbd468eaa29cc00 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Wed, 23 Mar 2022 09:23:08 +0100 Subject: [PATCH 24/27] Update docs/source/post_processing.md Co-authored-by: Samuel Farrens --- docs/source/post_processing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/post_processing.md b/docs/source/post_processing.md index d99fd8e50..a4e94918b 100644 --- a/docs/source/post_processing.md +++ b/docs/source/post_processing.md @@ -112,6 +112,6 @@ At this step all required `ShapePipe` resulting output files are available in th merge_final_cat -i -p -v ``` Choose as input directory `input_dir` the output of step C. A default - parameter file is `/path/to/shapepipe/example/cfis/final_cat.param`. + parameter file `` is `/path/to/shapepipe/example/cfis/final_cat.param`. On success, the file `./final_cat.npy` is created. Depending on the number of input tiles, this file can be several tens of Gb large. From b6d467f8e38d148f685360d8f70f1de3c6208dfd Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Wed, 23 Mar 2022 09:24:33 +0100 Subject: [PATCH 25/27] PR review changes to doc --- docs/source/post_processing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/post_processing.md b/docs/source/post_processing.md index d99fd8e50..b35f4d15b 100644 --- a/docs/source/post_processing.md +++ b/docs/source/post_processing.md @@ -5,9 +5,9 @@ more `ShapePipe` runs. Post-processing combines various individual 'ShapePipe' output files, and creates joint results, for example combining individual tile catalogues in a large sky area. The output of post-processing is a joint _shape catalogue_, containing all required information to create a calibrated shear -catalogue via 'metacalibration'), a joint star catalogue, and PSF diagnostic plots. +catalogue via _metacalibration_), a joint star catalogue, and PSF diagnostic plots. -Some of the following steps pertain specifically to runs carried out on [canfar](./canfar.md), +Some of the following steps pertain specifically to runs carried out on [canfar](https://www.canfar.net/en), but most are general. 1. Retrieve `ShapePipe` result files From 5dc951b9dba6b6f7b6ded36ce16620b0f0b16718 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Wed, 23 Mar 2022 09:26:19 +0100 Subject: [PATCH 26/27] PR review changes to doc --- shapepipe/modules/mccd_package/__init__.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/shapepipe/modules/mccd_package/__init__.py b/shapepipe/modules/mccd_package/__init__.py index 408e74c33..e42975319 100644 --- a/shapepipe/modules/mccd_package/__init__.py +++ b/shapepipe/modules/mccd_package/__init__.py @@ -145,7 +145,7 @@ This module generates different series of plots of the input merged star catalogue. -One series of plots are the "mean shapes", which is the PSF ellipticity and size +One series of plots are the _mean shapes_, which is the PSF ellipticity and size and their residuals as a function of the focal plane positions. Values and residuals are combined from all merged catalogues, and binned in the focal plane according to ``X_GRID`` and ``Y_GRID``. @@ -162,15 +162,15 @@ PSF: str PSF model type, options are ``mccd`` or ``psfex`` PLOT_MEANSHAPES: bool - Option to produce mean shapes plots + Option to produce _mean shapes_ plots X_GRID: int - Number of bins in the X axis of one CCD for the mean shapes plot grid + Number of bins in the X axis of one CCD for the _mean shapes_ plot grid Y_GRID: int - Number of bins in the Y axis of one CCD for the mean shapes plot grid + Number of bins in the Y axis of one CCD for the _mean shapes_ plot grid MAX_E: float - Max value for ellipticity in the mean shapes plot + Max value for ellipticity in the _mean shapes_ plot MAX_DE: float - Max value for the residual ellipticity in the mean shapes plot + Max value for the residual ellipticity in the _mean shapes_ plot PLOT_HISTOGRAMS: bool Option to produce histogram plots for shape errors REMOVE_OUTLIERS: bool From 367ba306db67403dac1afbec9e03539e63e9e4c8 Mon Sep 17 00:00:00 2001 From: Martin Kilbinger Date: Wed, 23 Mar 2022 09:50:54 +0100 Subject: [PATCH 27/27] Update docs/source/post_processing.md Co-authored-by: Samuel Farrens --- docs/source/post_processing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/post_processing.md b/docs/source/post_processing.md index 60c35b778..d3edec0ea 100644 --- a/docs/source/post_processing.md +++ b/docs/source/post_processing.md @@ -1,7 +1,7 @@ # Post-processing This page shows all required steps of post-processing the results from one or -more `ShapePipe` runs. Post-processing combines various individual 'ShapePipe' +more `ShapePipe` runs. Post-processing combines various individual `ShapePipe` output files, and creates joint results, for example combining individual tile catalogues in a large sky area. The output of post-processing is a joint _shape catalogue_, containing all required information to create a calibrated shear